2024-03-31T15:38:03.553 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-31T15:38:03.553 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/pool-perm} --verbose --owner scheduled_vshankar@teuthology --name vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi -- /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/orig.config.yaml 2024-03-31T15:38:03.597 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-31T15:38:03.687 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664 branch: wip-vshankar-testing-20240330.170739 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/pool-perm} email: vshankar@redhat.com first_in_suite: false job_id: '7631664' kernel: flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20240330.170739 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240330.170739 sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6891 sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 sleep_before_teardown: 0 subset: 932/1024 suite: fs suite_branch: wip-vshankar-testing-20240330.170739 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKckH9MSjojHRMLXVlBU2myYnNZCRAuBqTRWv9TLz664o+HCyz7XYHhQg/z6xP5TlufTtSBUbEPqjadTUuGjeXM= smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBb/CxZGUrMOZXbVtwaG8bt0zun6ADwzwAEZXPWOWexVynz8YWDsB3qZv8Z66DgK2HJsZ1/00hqYcnbT8T33ugM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-30_23:12:08 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-31T15:38:03.687 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa; will attempt to use it 2024-03-31T15:38:03.688 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks 2024-03-31T15:38:03.688 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-31T15:38:03.690 INFO:teuthology.task.internal:Checking packages... 2024-03-31T15:38:03.715 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2271579d5e735cfd6d436b8349c004f06527cb0' 2024-03-31T15:38:03.716 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-31T15:38:03.716 INFO:teuthology.packaging:ref: None 2024-03-31T15:38:03.716 INFO:teuthology.packaging:tag: None 2024-03-31T15:38:03.716 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240330.170739 2024-03-31T15:38:03.716 INFO:teuthology.packaging:sha1: d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:38:03.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20240330.170739 2024-03-31T15:38:03.940 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:38:03.942 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-31T15:38:03.949 INFO:teuthology.task.internal:no buildpackages task found 2024-03-31T15:38:03.949 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-31T15:38:03.965 INFO:teuthology.task.internal:Saving configuration 2024-03-31T15:38:03.982 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-31T15:38:03.988 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T15:38:04.014 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 15:31:39.299439', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDfgD/825pBA9OqXDgzBiRlQCvRVzBUQofLnf5YwxIlS'} 2024-03-31T15:38:04.037 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 15:31:39.301092', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIT7yi5MgNaqZAwbqGG84YLEQD9ok+VRbhST6PgY9Sxd'} 2024-03-31T15:38:04.037 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-31T15:38:04.044 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-31T15:38:04.044 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-31T15:38:04.044 INFO:teuthology.run_tasks:Running task console_log... 2024-03-31T15:38:04.131 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c4720a4c0>, signals=[15]) 2024-03-31T15:38:04.131 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-31T15:38:04.138 INFO:teuthology.task.internal:Opening connections... 2024-03-31T15:38:04.138 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:38:04.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:38:04.220 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:38:04.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:38:04.300 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-31T15:38:04.307 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-03-31T15:38:04.312 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-03-31T15:38:04.312 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-03-31T15:38:04.358 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="22.04" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=jammy 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-31T15:38:04.359 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=jammy 2024-03-31T15:38:04.360 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-03-31T15:38:04.383 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-31T15:38:04.388 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-31T15:38:04.388 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-31T15:38:04.434 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-31T15:38:04.434 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="Ubuntu" 2024-03-31T15:38:04.434 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="22.04" 2024-03-31T15:38:04.434 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_CODENAME=jammy 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:ID=ubuntu 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE=debian 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-31T15:38:04.435 INFO:teuthology.orchestra.run.smithi102.stdout:UBUNTU_CODENAME=jammy 2024-03-31T15:38:04.435 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-31T15:38:04.466 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-31T15:38:04.476 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-31T15:38:04.493 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-31T15:38:04.493 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-03-31T15:38:04.496 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-31T15:38:04.501 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-31T15:38:04.507 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-31T15:38:04.507 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-03-31T15:38:04.540 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-31T15:38:04.551 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-31T15:38:04.607 INFO:teuthology.run_tasks:Running task kernel... 2024-03-31T15:38:04.621 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-31T15:38:04.621 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, timeout 300 2024-03-31T15:38:04.622 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T15:38:04.622 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-31T15:38:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:38:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:38:04.628 DEBUG:teuthology.task.kernel:sha1 for mon.b is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.628 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-31T15:38:04.628 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-31T15:38:04.628 DEBUG:teuthology.task.kernel:sha1 for mon.a is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.628 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-31T15:38:04.629 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-31T15:38:04.675 INFO:teuthology.orchestra.run.smithi102.stdout:5.15.0-56-generic 2024-03-31T15:38:04.675 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.675 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-31T15:38:04.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.679 INFO:teuthology.orchestra.run.smithi090.stdout:5.15.0-56-generic 2024-03-31T15:38:04.679 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.680 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-31T15:38:04.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:38:04.820 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-03-31T15:38:04.820 DEBUG:teuthology.orchestra.run.smithi102:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-31T15:38:04.838 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-31T15:38:04.902 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-03-31T15:38:04.903 DEBUG:teuthology.orchestra.run.smithi090:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-31T15:38:04.918 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-31T15:38:04.978 INFO:teuthology.orchestra.run.smithi102.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-31T15:38:04.978 INFO:teuthology.orchestra.run.smithi102.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: git config --global init.defaultBranch 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: 2024-03-31T15:38:04.979 INFO:teuthology.orchestra.run.smithi102.stderr:hint: git branch -m 2024-03-31T15:38:04.980 INFO:teuthology.orchestra.run.smithi102.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-31T15:38:04.982 DEBUG:teuthology.orchestra.run.smithi102:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: git config --global init.defaultBranch 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: 2024-03-31T15:38:05.035 INFO:teuthology.orchestra.run.smithi090.stderr:hint: git branch -m 2024-03-31T15:38:05.036 INFO:teuthology.orchestra.run.smithi090.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-31T15:38:05.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-31T15:38:05.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-31T15:38:05.106 DEBUG:teuthology.orchestra.run.smithi090:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-31T15:38:05.112 DEBUG:teuthology.orchestra.run.smithi102:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-31T15:38:05.161 DEBUG:teuthology.orchestra.run.smithi090:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-31T15:39:19.987 INFO:teuthology.orchestra.run.smithi090.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-31T15:39:19.987 INFO:teuthology.orchestra.run.smithi090.stderr: * [new branch] main -> origin/main 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190312 -> 20190312 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190416 -> 20190416 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190514 -> 20190514 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190618 -> 20190618 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190717 -> 20190717 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190815 -> 20190815 2024-03-31T15:39:20.014 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20190923 -> 20190923 2024-03-31T15:39:20.015 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20191022 -> 20191022 2024-03-31T15:39:20.015 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20191215 -> 20191215 2024-03-31T15:39:20.017 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200122 -> 20200122 2024-03-31T15:39:20.017 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200316 -> 20200316 2024-03-31T15:39:20.017 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200421 -> 20200421 2024-03-31T15:39:20.017 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200519 -> 20200519 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200619 -> 20200619 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200721 -> 20200721 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200817 -> 20200817 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20200918 -> 20200918 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20201022 -> 20201022 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20201118 -> 20201118 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20201218 -> 20201218 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210208 -> 20210208 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210315 -> 20210315 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210511 -> 20210511 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210716 -> 20210716 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210818 -> 20210818 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20210919 -> 20210919 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20211027 -> 20211027 2024-03-31T15:39:20.018 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20211216 -> 20211216 2024-03-31T15:39:20.019 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220209 -> 20220209 2024-03-31T15:39:20.019 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220310 -> 20220310 2024-03-31T15:39:20.019 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220411 -> 20220411 2024-03-31T15:39:20.019 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220509 -> 20220509 2024-03-31T15:39:20.019 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220610 -> 20220610 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220708 -> 20220708 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220815 -> 20220815 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20220913 -> 20220913 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20221012 -> 20221012 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20221109 -> 20221109 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20221214 -> 20221214 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230117 -> 20230117 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230210 -> 20230210 2024-03-31T15:39:20.021 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230310 -> 20230310 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230404 -> 20230404 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230515 -> 20230515 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230625 -> 20230625 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230804 -> 20230804 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20230919 -> 20230919 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20231030 -> 20231030 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20231111 -> 20231111 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20231211 -> 20231211 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20240115 -> 20240115 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20240220 -> 20240220 2024-03-31T15:39:20.022 INFO:teuthology.orchestra.run.smithi090.stderr: * [new tag] 20240312 -> 20240312 2024-03-31T15:39:21.779 INFO:teuthology.orchestra.run.smithi102.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-31T15:39:21.779 INFO:teuthology.orchestra.run.smithi102.stderr: * [new branch] main -> origin/main 2024-03-31T15:39:21.806 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190312 -> 20190312 2024-03-31T15:39:21.806 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190416 -> 20190416 2024-03-31T15:39:21.806 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190514 -> 20190514 2024-03-31T15:39:21.807 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190618 -> 20190618 2024-03-31T15:39:21.807 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190717 -> 20190717 2024-03-31T15:39:21.808 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190815 -> 20190815 2024-03-31T15:39:21.808 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20190923 -> 20190923 2024-03-31T15:39:21.808 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20191022 -> 20191022 2024-03-31T15:39:21.808 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20191215 -> 20191215 2024-03-31T15:39:21.808 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200122 -> 20200122 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200316 -> 20200316 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200421 -> 20200421 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200519 -> 20200519 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200619 -> 20200619 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200721 -> 20200721 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200817 -> 20200817 2024-03-31T15:39:21.809 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20200918 -> 20200918 2024-03-31T15:39:21.810 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20201022 -> 20201022 2024-03-31T15:39:21.810 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20201118 -> 20201118 2024-03-31T15:39:21.810 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20201218 -> 20201218 2024-03-31T15:39:21.810 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210208 -> 20210208 2024-03-31T15:39:21.811 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210315 -> 20210315 2024-03-31T15:39:21.811 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210511 -> 20210511 2024-03-31T15:39:21.811 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210716 -> 20210716 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210818 -> 20210818 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20210919 -> 20210919 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20211027 -> 20211027 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20211216 -> 20211216 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220209 -> 20220209 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220310 -> 20220310 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220411 -> 20220411 2024-03-31T15:39:21.812 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220509 -> 20220509 2024-03-31T15:39:21.813 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220610 -> 20220610 2024-03-31T15:39:21.813 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220708 -> 20220708 2024-03-31T15:39:21.813 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220815 -> 20220815 2024-03-31T15:39:21.814 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20220913 -> 20220913 2024-03-31T15:39:21.814 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20221012 -> 20221012 2024-03-31T15:39:21.814 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20221109 -> 20221109 2024-03-31T15:39:21.814 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20221214 -> 20221214 2024-03-31T15:39:21.814 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230117 -> 20230117 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230210 -> 20230210 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230310 -> 20230310 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230404 -> 20230404 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230515 -> 20230515 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230625 -> 20230625 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230804 -> 20230804 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20230919 -> 20230919 2024-03-31T15:39:21.815 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20231030 -> 20231030 2024-03-31T15:39:21.816 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20231111 -> 20231111 2024-03-31T15:39:21.816 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20231211 -> 20231211 2024-03-31T15:39:21.816 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20240115 -> 20240115 2024-03-31T15:39:21.816 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20240220 -> 20240220 2024-03-31T15:39:21.817 INFO:teuthology.orchestra.run.smithi102.stderr: * [new tag] 20240312 -> 20240312 2024-03-31T15:39:24.224 INFO:teuthology.orchestra.run.smithi090.stderr:Updating files: 50% (1831/3598) Updating files: 51% (1835/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 64% (2308/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 77% (2785/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 86% (3113/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-31T15:39:24.229 INFO:teuthology.orchestra.run.smithi090.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-31T15:39:24.250 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-31T15:39:24.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:39:24.490 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ 2024-03-31T15:39:24.490 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ --input-file=- 2024-03-31T15:39:24.493 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi090.front.sepia.ceph.com to complete... 2024-03-31T15:39:26.024 INFO:teuthology.orchestra.run.smithi102.stderr:Updating files: 51% (1848/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 64% (2323/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 77% (2785/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 86% (3113/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-31T15:39:26.029 INFO:teuthology.orchestra.run.smithi102.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-31T15:39:26.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31 15:39:26 URL:https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb [39278344/39278344] -> "/tmp/linux-image.deb" [1] 2024-03-31T15:39:26.045 INFO:teuthology.orchestra.run.smithi090.stderr:FINISHED --2024-03-31 15:39:26-- 2024-03-31T15:39:26.045 INFO:teuthology.orchestra.run.smithi090.stderr:Total wall clock time: 1.5s 2024-03-31T15:39:26.045 INFO:teuthology.orchestra.run.smithi090.stderr:Downloaded: 1 files, 37M in 1.4s (27.3 MB/s) 2024-03-31T15:39:26.049 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.a... 2024-03-31T15:39:26.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo dpkg -i /tmp/linux-image.deb 2024-03-31T15:39:26.051 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-31T15:39:26.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-31T15:39:26.198 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ 2024-03-31T15:39:26.199 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/ --input-file=- 2024-03-31T15:39:26.202 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-03-31T15:39:26.326 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-image-6.8.0-g1df0c7e8a235. 2024-03-31T15:39:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31 15:39:28 URL:https://4.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g1df0c7e8a235/linux-image-6.8.0-g1df0c7e8a235_6.8.0-g1df0c7e8a235-1_amd64.deb [39278344/39278344] -> "/tmp/linux-image.deb" [1] 2024-03-31T15:39:27.927 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2024-03-31 15:39:28-- 2024-03-31T15:39:27.927 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 1.7s 2024-03-31T15:39:27.927 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 37M in 1.5s (25.5 MB/s) 2024-03-31T15:39:27.931 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on mon.b... 2024-03-31T15:39:27.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo dpkg -i /tmp/linux-image.deb 2024-03-31T15:39:28.188 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-image-6.8.0-g1df0c7e8a235. 2024-03-31T15:39:28.999 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-31T15:39:29.005 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-31T15:39:29.147 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T15:39:30.253 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-31T15:39:30.257 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-31T15:39:30.373 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T15:39:33.640 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T15:39:33.706 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-image-6.8.0-g1df0c7e8a235 (6.8.0-g1df0c7e8a235-1) ... 2024-03-31T15:39:33.779 INFO:teuthology.orchestra.run.smithi090.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:39:33.833 INFO:teuthology.orchestra.run.smithi102.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:39:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub' 2024-03-31T15:39:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T15:39:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-31T15:39:54.003 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:39:54.024 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:39:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T15:39:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T15:39:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T15:39:54.378 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T15:39:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T15:39:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T15:39:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T15:39:54.569 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-31T15:39:54.732 DEBUG:teuthology.orchestra.run.smithi102:> dpkg-deb -c /tmp/linux-image.deb 2024-03-31T15:39:54.739 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./ 2024-03-31T15:39:54.739 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./boot/ 2024-03-31T15:39:54.740 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 4474334 2024-03-25 03:29 ./boot/System.map-6.8.0-g1df0c7e8a235 2024-03-31T15:39:54.752 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 170899 2024-03-25 03:29 ./boot/config-6.8.0-g1df0c7e8a235 2024-03-31T15:39:54.752 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11551744 2024-03-25 03:29 ./boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:39:54.771 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/ 2024-03-31T15:39:54.771 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postinst.d/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postrm.d/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/preinst.d/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/prerm.d/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ 2024-03-31T15:39:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 490441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-31T15:39:54.775 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-31T15:39:54.775 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-31T15:39:54.775 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 93073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 106689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 152137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 83849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-31T15:39:54.776 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/ 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/ 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/ 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpuid.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/msr.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/ 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 354225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-31T15:39:54.777 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 886129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-31T15:39:54.778 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 2294793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/af_alg.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_hash.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_skcipher.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-31T15:39:54.798 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/ 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/ 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-31T15:39:54.799 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cryptd.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_simd.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-31T15:39:54.800 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lrw.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-31T15:39:54.801 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 143025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xts.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_extlog.ko 2024-03-31T15:39:54.802 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/ 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/einj.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 104417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 89929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 88305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci.ko 2024-03-31T15:39:54.803 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-31T15:39:54.804 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_legacy.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-31T15:39:54.805 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_radisys.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rz1000.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-31T15:39:54.806 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 83249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-31T15:39:54.807 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 124385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 113801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 964745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 169697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 94209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 116329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/ 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 110321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 98185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-31T15:39:54.808 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 263633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/ 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 98105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/ 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/applicom.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/ 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-31T15:39:54.809 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 148513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 139305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-31T15:39:54.810 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/ 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 84001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/nvram.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/ 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-31T15:39:54.811 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/ 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/ 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-palmas.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-si5351.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-twl6040.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-wm831x.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/ 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-31T15:39:54.812 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 246777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/ 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/ 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-31T15:39:54.813 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 570217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-31T15:39:54.886 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-31T15:39:54.886 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 122793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-31T15:39:54.887 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/ 2024-03-31T15:39:54.888 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-gpio.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max14577.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max77693.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max8997.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-palmas.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/ 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/ 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/ 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-31T15:39:54.889 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-arizona.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9052.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9055.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-dln2.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 77745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-generic.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-kempld.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max7300.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max730x.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max732x.ko 2024-03-31T15:39:54.890 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-vx855.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-31T15:39:54.891 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/ 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-a4tech.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-apple.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-belkin.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cherry.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-chicony.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 60657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cp2112.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cypress.ko 2024-03-31T15:39:54.892 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ezkey.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-generic.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-31T15:39:54.893 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kensington.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lg-g15.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 57969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-magicmouse.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-microsoft.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-monterey.ko 2024-03-31T15:39:54.894 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ntrig.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-31T15:39:54.895 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-31T15:39:54.896 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 143377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-31T15:39:54.897 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 269209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/ 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 125625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 218865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/ 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 64297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 90089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 304913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-31T15:39:54.898 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 97881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-31T15:39:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 99505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-31T15:39:54.900 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc2103.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 94593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hih6130.ko 2024-03-31T15:39:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 140129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/jc42.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-31T15:39:54.902 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-31T15:39:54.903 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-31T15:39:54.904 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 119841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 82713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-31T15:39:54.905 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 82473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-31T15:39:54.906 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 43441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-31T15:39:54.907 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 79833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 71249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 63129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 71225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 119409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/ 2024-03-31T15:39:54.908 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/ 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/ 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-31T15:39:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-31T15:39:54.910 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-31T15:39:54.911 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/ 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 273513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 883777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 324233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-31T15:39:54.912 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 97825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 282329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 493129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 420529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 741609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 296785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 153705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-31T15:39:54.913 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 169089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 266529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 100257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 111329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 128657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 129793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ 2024-03-31T15:39:54.914 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/evbug.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ff-memless.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/ 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/ 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-31T15:39:54.915 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8323.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8333.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt1070.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt2160.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/stowaway.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-31T15:39:54.916 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/matrix-keymap.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/ 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 123177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 89377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 335857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/psmouse.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-31T15:39:54.917 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/ 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 109553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ct82c710.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/parkbd.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/pcips2.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ps2mult.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serport.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/ 2024-03-31T15:39:54.918 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-88pm860x.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-adp5520.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-bd2802.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da903x.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da9052.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-gpio.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3533.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm355x.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3642.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp8788.ko 2024-03-31T15:39:54.919 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-max8997.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-mc13783.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca9532.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca955x.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca963x.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-tca6507.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm8350.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-31T15:39:54.920 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/ 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/ 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 647033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 93313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bufio.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 155801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-31T15:39:54.921 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 107929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-era.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-mirror.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 77865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-region-hash.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-31T15:39:54.922 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 109433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-snapshot.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 181761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-zero.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/ 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 199825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 100833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 126361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 368481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/ 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/ 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 178041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-31T15:39:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 63233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 123225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/ 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm800.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm805.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm80x.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona-i2c.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 123625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/bcm590xx.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-31T15:39:54.924 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/dln2.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/janz-cmodio.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/kempld-core.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-core.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lp3943.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/max8907.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/menf21bmc.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-31T15:39:54.925 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/retu-mfd.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/si476x-core.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps65010.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps6507x.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wm8994.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot.ko 2024-03-31T15:39:54.926 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/ 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/ 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/core.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/ 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ds1682.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/dummy-irq.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/ 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/echo.ko 2024-03-31T15:39:54.927 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/ 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/ 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 152041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hmc6352.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ics932s401.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/ 2024-03-31T15:39:54.928 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/ 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 126137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 313545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/phantom.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/ 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/ 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 183513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-31T15:39:54.929 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 491561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 105929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 458905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/ 2024-03-31T15:39:54.930 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 130657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 68777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 64609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/ 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/ 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/ 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-31T15:39:54.931 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 87449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 97185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 102481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 90185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-31T15:39:54.932 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 81313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 176689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1434449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 128361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 131265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 314321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 323681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-31T15:39:54.933 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 193537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 347161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1045321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 166129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 204185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-31T15:39:54.934 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/ 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 141153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/ 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 60665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/ 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-31T15:39:54.935 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 281193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 90297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 295673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 584297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 245977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-31T15:39:54.936 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 944449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 288481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 514337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 93465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 735129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 151649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 86985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/ 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-31T15:39:54.937 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 102433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 122601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/ 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 727377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 278825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1122065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 95569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-31T15:39:54.938 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 151193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 136793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/ 2024-03-31T15:39:54.939 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 227841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 87009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 671985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 83801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 194057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 104001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-31T15:39:54.940 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1066817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 63193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 68041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-31T15:39:54.941 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-31T15:39:54.942 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-31T15:39:54.942 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-31T15:39:54.942 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 495641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/ 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 166041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/ 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/ 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 92337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/ 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 79929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-31T15:39:55.071 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 83713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/defxx.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 217025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 237241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/ 2024-03-31T15:39:55.072 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 85841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/ 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-31T15:39:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/ 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-31T15:39:55.074 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 81953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 76561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 99001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/ 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 43761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/plip.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-31T15:39:55.075 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sb1000.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/ 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 64553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/ 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 107225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-31T15:39:55.076 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 86817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/ 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 144345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/ 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 307169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/ 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 143993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-31T15:39:55.077 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 267721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 131297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 85209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/ 2024-03-31T15:39:55.078 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pci-stub.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/ 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 146129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 112785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/ 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/ 2024-03-31T15:39:55.079 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/ 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/ 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/ 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/ 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/ 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-31T15:39:55.080 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 131937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-31T15:39:55.081 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/ 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-31T15:39:55.082 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/ 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9030_battery.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9052-battery.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-31T15:39:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/gpio-charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max14577_charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max17042_battery.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8903_charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8925_power.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/sbs-battery.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/test_power.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-31T15:39:55.084 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_power.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm8350_power.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/ 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/ 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/ 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-31T15:39:55.085 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/ 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9052.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9055.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9063.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 68473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-31T15:39:55.086 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-31T15:39:55.087 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8907.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8925.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8997.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8998.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-palmas.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-31T15:39:55.088 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ 2024-03-31T15:39:55.089 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 77521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 57713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 135249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/ 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 271385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 100529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/ 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 356833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 295889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/ 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 189089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-31T15:39:55.090 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/ 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/ 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 246913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/ 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1171857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/ 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 238369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/ 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 115305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-31T15:39:55.091 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 308905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 76033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 146745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 75969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 212329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 60497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/ 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-31T15:39:55.092 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 157713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fdomain.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/ 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 294881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 201721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/imm.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 43969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 265937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/ 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 317129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-31T15:39:55.093 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/ 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 315273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 145489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/ 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 216345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/ 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 2193953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/ 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 310665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-31T15:39:55.094 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 85513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/ 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 659585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/ 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 119465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/ 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 81545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/ 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 366305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-31T15:39:55.095 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 124593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ppa.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/ 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1876657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 85185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/ 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 558897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qlogicfas408.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 261073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 173921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 280105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-31T15:39:55.096 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 97617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 115641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 164865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/xen-scsifront.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 129345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-31T15:39:55.097 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/ 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/ 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 736665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/ 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 931433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 122481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/ 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 75497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-31T15:39:55.098 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/ 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/ 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-31T15:39:55.099 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 57769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/moxa.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/mxser.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-31T15:39:55.100 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/ 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/ 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_uart.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/ 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 101969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/rp2.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-31T15:39:55.101 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/ 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 545377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/ 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_mf624.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_netx.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-31T15:39:55.102 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/ 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/ 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 145753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/ 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 87001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-31T15:39:55.103 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 89761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ulpi.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/ 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/ 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 452785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/ 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/ 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 60321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-31T15:39:55.104 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 92401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 50937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 122241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 57625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 85505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/ 2024-03-31T15:39:55.105 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-31T15:39:55.106 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 161193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/ 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 157593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 69609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 75961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 73009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-31T15:39:55.107 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 65873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 65993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 81401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 156321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 87249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811_cs.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/ 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-31T15:39:55.108 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cytherm.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ehset.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-31T15:39:55.109 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/lvstest.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/ 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 64737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbtest.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/ 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/usbmon.ko 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/ 2024-03-31T15:39:55.110 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 301593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/ 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-31T15:39:55.111 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/f81232.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 175097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 136977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 124737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 107425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-31T15:39:55.112 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/metro-usb.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 78121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mxuport.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 191817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-31T15:39:55.113 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 125513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usbserial.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-31T15:39:55.114 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 52753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-31T15:39:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 180281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/ 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 71001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/ 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/ 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 162841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 76433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/ 2024-03-31T15:39:55.116 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 113657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 57993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 89049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/ 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/ 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/bd6107.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-31T15:39:55.117 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/ 2024-03-31T15:39:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/ 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-31T15:39:55.119 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/acquirewdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/advantechwdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/dw_wdt.ko 2024-03-31T15:39:55.120 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-31T15:39:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/retu_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-31T15:39:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/ 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-31T15:39:55.123 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 176337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 71145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-scsiback.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 158217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/adfs.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 163369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/ 2024-03-31T15:39:55.124 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1019049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/kafs.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 103185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/autofs4.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 64585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 41001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/bfs.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 3487249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 305841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1335577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/ 2024-03-31T15:39:55.125 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 97713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/configfs.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 644161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/efs.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1527561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/ 2024-03-31T15:39:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/freevxfs.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1231233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 141625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 223177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 193481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/hpfs.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/ 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-31T15:39:55.127 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 420329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 281617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 88897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 490409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 81433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/ 2024-03-31T15:39:55.128 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 117361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1058273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv2.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 120169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 2041705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/ 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/ 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1511641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/ 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 582409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/ 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-31T15:39:55.129 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ascii.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-31T15:39:55.130 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 94033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-31T15:39:55.131 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 142569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 165641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 118017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-31T15:39:55.132 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 220457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ntfs.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 244633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 440633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 2590681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-31T15:39:55.133 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/omfs.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 360897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 56929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/qnx4.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 46601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/qnx6.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/ 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_tree.ko 2024-03-31T15:39:55.134 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v1.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v2.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 532161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 2712385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 134705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/ 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 91281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-31T15:39:55.135 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 248329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 162737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 4035417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 80481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/locktorture.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/test_udelay.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 60257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc7.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-31T15:39:55.136 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/ 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/interval_tree_test.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/ 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/ 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/memory-notifier-error-inject.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/notifier-error-inject.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/percpu_test.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/pm-notifier-error-inject.ko 2024-03-31T15:39:55.137 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/ 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 143257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/rbtree_test.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/ 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-kstrtox.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-string_helpers.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 4863217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_bpf.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_firmware.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_module.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_user_copy.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-31T15:39:55.138 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/ 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/xz_dec_test.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/ 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/ 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 6897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/ 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-31T15:39:55.139 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/ 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 82865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/ 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 198809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/ 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 667353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/ 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 790545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-31T15:39:55.140 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-31T15:39:55.141 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/ 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1046689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/ 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 125697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/ 2024-03-31T15:39:55.142 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 194441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 58737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/ 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 390033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/ 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 102033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/hsr.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-31T15:39:55.143 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 284665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 47873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 51209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 61505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-31T15:39:55.144 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-31T15:39:55.145 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-31T15:39:55.146 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 45041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-31T15:39:55.147 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 79921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_gre.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 94161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 54265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/mip6.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-31T15:39:55.148 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-31T15:39:55.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 74873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/ 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 84393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/ 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 125201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 25809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-31T15:39:55.150 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 53225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/lapb.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 268449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc2.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 309361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ 2024-03-31T15:39:55.151 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 119689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 99985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 100273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 102545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 102337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 111665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 59185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 112137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 112401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 110329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 113281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-31T15:39:55.152 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 412433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-31T15:39:55.153 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 411665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 118145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 108305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-31T15:39:55.154 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 92217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 96489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 44129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 623337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-31T15:39:55.155 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 42505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 22929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 40801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-31T15:39:55.156 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_osf.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_synproxy.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tproxy.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 89225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/x_tables.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-31T15:39:55.157 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 38769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-31T15:39:55.158 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-31T15:39:55.159 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 43193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-31T15:39:55.160 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-31T15:39:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 29993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 16809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 8449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpudp.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/ 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 14433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/ 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 12329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-31T15:39:55.162 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/ 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 384137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/ 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/ 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 112521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/phonet.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/pn_pep.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/ 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 253881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 165049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 68065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-31T15:39:55.163 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/ 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 828145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/rxrpc.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/ 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 37081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 43345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 19177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_cgroup.ko 2024-03-31T15:39:55.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 39513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 62649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 13313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 33417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 34585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-31T15:39:55.165 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq_codel.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 35449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 49841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 66441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 30217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 36145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 27401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 24769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 48529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 32809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-31T15:39:55.166 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 26785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 31313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 28265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/ 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 932785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/ 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/ 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 336969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 1513769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/ 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 704697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/ 2024-03-31T15:39:55.167 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 867121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/ 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/ 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 87737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 67937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 98089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 11073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/ 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 131209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/x25.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/ 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_algo.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 20545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-31T15:39:55.168 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 105209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_user.ko 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/ 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/ 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 15433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 642383 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 625850 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 7677 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 23498 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 9916 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 70732 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 147214 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 233213 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 210 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 72614 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-31T15:39:55.169 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 879 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 257955 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 312782 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/ 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/ 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/ 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/ 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 169 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/changelog.Debian.gz 2024-03-31T15:39:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r-- root/root 691 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/copyright 2024-03-31T15:39:55.175 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-31T15:39:55.175 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g1df0c7e8a235 2024-03-31T15:39:55.175 DEBUG:teuthology.orchestra.run.smithi102:> egrep '(submenu|menuentry.*6.8.0-g1df0c7e8a235).*{' /boot/grub/grub.cfg 2024-03-31T15:39:55.182 INFO:teuthology.orchestra.run.smithi102.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:39:55.182 INFO:teuthology.orchestra.run.smithi102.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:39:55.182 INFO:teuthology.orchestra.run.smithi102.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:39:55.183 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-31T15:39:55.183 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g1df0c7e8a235 2024-03-31T15:39:55.183 DEBUG:teuthology.orchestra.run.smithi102:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-31T15:39:55.227 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-03-31T15:39:56.038 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub' 2024-03-31T15:39:56.041 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T15:39:56.052 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-31T15:39:56.243 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:39:56.249 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:39:56.546 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T15:39:56.552 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T15:39:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T15:39:56.598 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T15:39:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T15:39:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T15:39:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T15:39:56.727 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-31T15:39:59.915 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub' 2024-03-31T15:39:59.945 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T15:39:59.980 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-03-31T15:40:00.377 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:40:00.420 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:40:01.031 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T15:40:01.050 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T15:40:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T15:40:01.136 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T15:40:01.417 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T15:40:01.417 INFO:teuthology.orchestra.run.smithi090.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T15:40:01.417 INFO:teuthology.orchestra.run.smithi090.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T15:40:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-03-31T15:40:01.713 DEBUG:teuthology.orchestra.run.smithi090:> dpkg-deb -c /tmp/linux-image.deb 2024-03-31T15:40:01.722 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./ 2024-03-31T15:40:01.722 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./boot/ 2024-03-31T15:40:01.722 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 4474334 2024-03-25 03:29 ./boot/System.map-6.8.0-g1df0c7e8a235 2024-03-31T15:40:01.734 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 170899 2024-03-25 03:29 ./boot/config-6.8.0-g1df0c7e8a235 2024-03-31T15:40:01.735 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11551744 2024-03-25 03:29 ./boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:40:01.754 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/ 2024-03-31T15:40:01.754 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/ 2024-03-31T15:40:01.754 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postinst.d/ 2024-03-31T15:40:01.754 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/postrm.d/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/preinst.d/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./etc/kernel/prerm.d/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ 2024-03-31T15:40:01.755 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 490441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 93073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 106689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 152137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-31T15:40:01.758 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 83849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/ 2024-03-31T15:40:01.759 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/ 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/ 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpuid.ko 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/msr.ko 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/ 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 354225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 886129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-31T15:40:01.760 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 2294793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-31T15:40:01.839 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/af_alg.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_hash.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/algif_skcipher.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/ 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/ 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-31T15:40:01.840 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-31T15:40:01.841 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cryptd.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_simd.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lrw.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-31T15:40:01.842 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 143025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-31T15:40:01.843 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xts.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_extlog.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/ 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/apei/einj.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 104417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-31T15:40:01.844 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 89929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 88305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-31T15:40:01.845 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_legacy.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-31T15:40:01.846 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_radisys.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rz1000.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-31T15:40:01.847 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 83249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-31T15:40:01.848 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 124385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 113801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 964745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 169697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/ 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 94209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 116329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-31T15:40:01.849 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/ 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 110321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 98185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 263633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/ 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 98105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/ 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/applicom.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/ 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-31T15:40:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 148513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 139305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/ 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 84001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/nvram.ko 2024-03-31T15:40:01.851 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/ 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/ 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-31T15:40:01.852 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/ 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-palmas.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-si5351.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-twl6040.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/clk/clk-wm831x.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/ 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 246777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/ 2024-03-31T15:40:01.853 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 570217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/ 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-31T15:40:01.854 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 122793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-31T15:40:01.855 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/ 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-gpio.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max14577.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max77693.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-max8997.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-palmas.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/ 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/ 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-31T15:40:01.856 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/ 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-arizona.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9052.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-da9055.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-dln2.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 77745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-generic.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-kempld.ko 2024-03-31T15:40:01.857 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max7300.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max730x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-max732x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-31T15:40:01.858 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-vx855.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/ 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-a4tech.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-apple.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-belkin.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cherry.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-chicony.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 60657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cp2112.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-cypress.ko 2024-03-31T15:40:01.859 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ezkey.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-generic.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-31T15:40:01.860 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kensington.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lg-g15.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 57969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-magicmouse.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-microsoft.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-monterey.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-31T15:40:01.861 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ntrig.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-31T15:40:01.862 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-31T15:40:01.863 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 143377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 269209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/ 2024-03-31T15:40:01.864 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 125625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 218865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/ 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 64297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 90089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 304913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-31T15:40:01.865 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 97881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-31T15:40:01.866 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 99505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc2103.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-31T15:40:01.867 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 94593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hih6130.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-31T15:40:01.868 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 140129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/jc42.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-31T15:40:01.869 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-31T15:40:01.870 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 119841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 82713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-31T15:40:01.871 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 82473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-31T15:40:01.872 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-31T15:40:01.873 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 43441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 79833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 71249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 63129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 71225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 119409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-31T15:40:01.874 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/ 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/ 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/ 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-31T15:40:01.875 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-31T15:40:01.876 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-31T15:40:01.877 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/ 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 273513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 883777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 324233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-31T15:40:01.878 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 97825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 282329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 493129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 420529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 741609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 296785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 153705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-31T15:40:01.879 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 169089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 266529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 100257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 111329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 128657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 129793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ 2024-03-31T15:40:01.880 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/evbug.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/ff-memless.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/ 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/ 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-31T15:40:01.881 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8323.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/lm8333.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt1070.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/qt2160.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/stowaway.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-31T15:40:01.882 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/matrix-keymap.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/ 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 123177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 89377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 335857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/psmouse.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-31T15:40:01.883 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/ 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 109553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ct82c710.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/parkbd.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/pcips2.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/ps2mult.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serport.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-31T15:40:01.884 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/ 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-88pm860x.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-adp5520.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-bd2802.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da903x.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-da9052.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-gpio.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3533.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm355x.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3642.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp8788.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-max8997.ko 2024-03-31T15:40:01.885 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-mc13783.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca9532.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca955x.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-pca963x.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-tca6507.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-wm8350.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-31T15:40:01.886 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/ 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/ 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 647033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 93313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bufio.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 155801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 107929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-era.ko 2024-03-31T15:40:01.887 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-mirror.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 77865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-region-hash.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 109433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-snapshot.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 181761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-zero.ko 2024-03-31T15:40:01.888 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/ 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 199825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 100833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 126361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 368481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/ 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/ 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 178041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 63233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 123225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-31T15:40:01.889 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/ 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm800.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm805.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/88pm80x.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona-i2c.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 123625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/arizona.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/bcm590xx.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/dln2.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/janz-cmodio.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/kempld-core.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-core.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-31T15:40:01.890 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lp3943.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/max8907.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/menf21bmc.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/pcf50633.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/retu-mfd.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/si476x-core.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps65010.ko 2024-03-31T15:40:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/tps6507x.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wm8994.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ad525x_dpot.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/ 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/ 2024-03-31T15:40:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/c2port/core.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/ 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ds1682.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/dummy-irq.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/ 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/echo/echo.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/ 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/ 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 152041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-31T15:40:01.893 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hmc6352.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ics932s401.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/ 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/ 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 126137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 313545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/phantom.ko 2024-03-31T15:40:01.894 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 183513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 491561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/ 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 105929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-31T15:40:01.895 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 458905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/ 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 130657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 68777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 64609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-31T15:40:01.896 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 87449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-31T15:40:01.897 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 97185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 102481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 90185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 81313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-31T15:40:01.898 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/ 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 176689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1434449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-31T15:40:01.982 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 128361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 131265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 314321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 323681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 193537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 347161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-31T15:40:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1045321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 166129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 204185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 141153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-31T15:40:01.984 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/ 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 60665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/ 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 281193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/ 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ 2024-03-31T15:40:01.985 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 90297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 295673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 584297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 245977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 944449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 288481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/ 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 514337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-31T15:40:01.986 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 93465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 735129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 151649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 86985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 102433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 122601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 727377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-31T15:40:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 278825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1122065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 95569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-31T15:40:01.988 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 151193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 136793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 227841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 87009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 671985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/ 2024-03-31T15:40:01.989 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/ 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 83801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 194057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/ 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 104001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/ 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1066817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/ 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-31T15:40:01.990 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/ 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 63193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/ 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 68041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/ 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 495641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-31T15:40:01.991 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/ 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 166041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/ 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/ 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 92337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/ 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 79929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 83713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/ 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-31T15:40:01.992 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/defxx.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 217025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 237241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 85841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-31T15:40:01.993 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/ 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/ 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-31T15:40:01.994 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 81953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 76561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-31T15:40:01.995 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 99001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/ 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 43761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/plip/plip.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-31T15:40:01.996 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sb1000.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/ 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 64553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/ 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 107225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-31T15:40:01.997 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 86817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 144345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 307169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 143993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/ 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 267721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-31T15:40:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/ 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 131297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 85209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/ 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/ 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pci-stub.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/ 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 146129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia.ko 2024-03-31T15:40:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 112785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/ 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-31T15:40:02.000 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 131937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-31T15:40:02.001 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/ 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-31T15:40:02.002 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/ 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-31T15:40:02.003 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9030_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/da9052-battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/gpio-charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max14577_charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max17042_battery.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8903_charger.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/max8925_power.ko 2024-03-31T15:40:02.004 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/sbs-battery.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/test_power.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm831x_power.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/power/supply/wm8350_power.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/ 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/ 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/ 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-31T15:40:02.005 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/ 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9052.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9055.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-da9063.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 68473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-31T15:40:02.006 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-31T15:40:02.007 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8907.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8925.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8997.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max8998.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-palmas.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-31T15:40:02.008 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-31T15:40:02.009 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 77521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 57713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 135249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/ 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 271385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 100529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/ 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 356833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 295889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-31T15:40:02.010 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 189089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 246913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1171857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 238369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/ 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 115305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-31T15:40:02.011 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 308905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 76033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 146745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 75969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 212329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 60497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/ 2024-03-31T15:40:02.012 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 157713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fdomain.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/ 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 294881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 201721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/imm.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 43969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 265937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/ 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 317129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-31T15:40:02.013 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/ 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 315273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 145489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/ 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 216345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/ 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 2193953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/ 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 310665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 85513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-31T15:40:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/ 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 659585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/ 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 119465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/ 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 81545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/ 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 366305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 124593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ppa.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-31T15:40:02.015 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/ 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1876657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 85185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/ 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 558897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qlogicfas408.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 261073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 173921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 280105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 97617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 115641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-31T15:40:02.016 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 164865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/xen-scsifront.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 129345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/ 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/ 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 736665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/ 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-31T15:40:02.017 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 931433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 122481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/ 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 75497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/ 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/ 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-31T15:40:02.018 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ 2024-03-31T15:40:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 57769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/moxa.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/mxser.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/ 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/ 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/altera_uart.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-31T15:40:02.020 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/ 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 101969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/rp2.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/ 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 545377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/ 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-31T15:40:02.021 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_mf624.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_netx.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/ 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/ 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 145753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/ 2024-03-31T15:40:02.022 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 87001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 89761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/common/ulpi.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/ 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/ 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 452785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/ 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/ 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 60321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-31T15:40:02.023 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 92401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 50937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 122241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 57625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 85505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-31T15:40:02.024 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/ 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-31T15:40:02.025 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 161193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/ 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 157593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 69609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 75961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 73009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-31T15:40:02.026 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 65873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 65993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 81401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 156321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 87249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811_cs.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/ 2024-03-31T15:40:02.027 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/cytherm.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ehset.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-31T15:40:02.028 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/lvstest.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/ 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 64737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbtest.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/ 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/mon/usbmon.ko 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/ 2024-03-31T15:40:02.029 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 301593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/ 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-31T15:40:02.030 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/f81232.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 175097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 136977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 124737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 107425 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-31T15:40:02.031 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/metro-usb.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 78121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mxuport.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 191817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-31T15:40:02.032 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 125513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usbserial.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ 2024-03-31T15:40:02.033 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 52753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 180281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-31T15:40:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/ 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 71001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/ 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/ 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 162841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 76433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/ 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 113657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 57993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-31T15:40:02.035 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 89049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/ 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/ 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/bd6107.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-31T15:40:02.036 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/ 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-31T15:40:02.037 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/ 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-31T15:40:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/acquirewdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/advantechwdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/dw_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-31T15:40:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-31T15:40:02.040 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/retu_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21473 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-31T15:40:02.041 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/ 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/ 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 176337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-31T15:40:02.042 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 71145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-scsiback.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 158217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/adfs/adfs.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 163369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1019049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/afs/kafs.ko 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/ 2024-03-31T15:40:02.043 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 103185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/autofs/autofs4.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 64585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 41001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/bfs/bfs.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 3487249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 305841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1335577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/ 2024-03-31T15:40:02.044 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 97713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/ 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/configfs/configfs.ko 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/ 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/ 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 644161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/ 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/efs/efs.ko 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/ 2024-03-31T15:40:02.045 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1527561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/freevxfs/freevxfs.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1231233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 141625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 223177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/ 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 193481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hpfs/hpfs.ko 2024-03-31T15:40:02.046 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 420329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 281617 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 88897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 490409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 81433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/ 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-31T15:40:02.047 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/ 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 117361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1058273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv2.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 120169 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 2041705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/ 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/ 2024-03-31T15:40:02.048 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1511641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-31T15:40:02.100 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/ 2024-03-31T15:40:02.100 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 582409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-31T15:40:02.100 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/ 2024-03-31T15:40:02.100 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ascii.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-31T15:40:02.101 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-31T15:40:02.102 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 94033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 142569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 165641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 118017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-31T15:40:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 220457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ntfs/ntfs.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 244633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 440633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-31T15:40:02.104 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 2590681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53681 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/omfs/omfs.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 360897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 56929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx4/qnx4.ko 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/ 2024-03-31T15:40:02.105 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 46601 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/qnx6/qnx6.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_tree.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v1.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/quota/quota_v2.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 532161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 2712385 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/ 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-31T15:40:02.106 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 134705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 91281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 248329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 162737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 4035417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 80481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/locking/locktorture.ko 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/ 2024-03-31T15:40:02.107 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/time/test_udelay.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 60257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc7.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/ 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/interval_tree_test.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/ 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-31T15:40:02.108 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/ 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/memory-notifier-error-inject.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/notifier-error-inject.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/percpu_test.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/pm-notifier-error-inject.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/ 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 143257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/rbtree_test.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/ 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-31T15:40:02.109 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-kstrtox.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test-string_helpers.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 4863217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_bpf.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_firmware.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_module.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/test_user_copy.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/ 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/xz/xz_dec_test.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/ 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ 2024-03-31T15:40:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/ 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6953 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 6897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/ 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-31T15:40:02.111 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/ 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 82865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/ 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 198809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/ 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 667353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/ 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 790545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ 2024-03-31T15:40:02.112 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-31T15:40:02.113 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/ 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1046689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/ 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 125697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/ 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 194441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-31T15:40:02.114 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 58737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/ 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 390033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12593 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/ 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 102033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/hsr/hsr.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 284665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 47873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-31T15:40:02.115 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 51209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 61505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-31T15:40:02.116 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9577 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14777 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-31T15:40:02.117 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15961 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-31T15:40:02.118 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10849 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 45041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-31T15:40:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 79921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_gre.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 94161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 54265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/mip6.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14033 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16105 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-31T15:40:02.120 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15017 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-31T15:40:02.121 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8905 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 74873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/ 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 84393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/ 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 125201 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 25809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-31T15:40:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 53225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48097 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/lapb/lapb.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 268449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc2.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 309361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 119689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28609 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-31T15:40:02.123 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 99985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 100273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 102545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 102337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 111665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 59185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 112137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 112401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 110329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 113281 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ 2024-03-31T15:40:02.124 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 412433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11985 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10633 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9257 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13537 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7713 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-31T15:40:02.125 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 411665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15873 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 118145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 108305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17289 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-31T15:40:02.126 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 92217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 96489 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11753 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 44129 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 623337 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 42505 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30153 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-31T15:40:02.127 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 22929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49641 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 40801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9497 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20025 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14321 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-31T15:40:02.128 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11745 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_osf.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9977 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12833 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_synproxy.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tproxy.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 89225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/x_tables.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7625 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-31T15:40:02.129 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 18073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13273 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 38769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13793 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11553 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-31T15:40:02.130 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8225 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11249 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12297 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-31T15:40:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7761 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10561 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 43193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11817 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11689 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15945 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7897 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-31T15:40:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10113 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11721 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13729 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10001 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11889 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9377 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8161 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-31T15:40:02.133 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36353 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13233 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 29993 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 16809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9409 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8705 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 8449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15673 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpudp.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9305 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/ 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 14433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/ 2024-03-31T15:40:02.134 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 12329 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/ 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 384137 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10361 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11825 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/ 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13465 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/ 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 112521 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/phonet.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55041 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/phonet/pn_pep.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/ 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 253881 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-31T15:40:02.135 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 165049 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 68065 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/ 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 828145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rxrpc/rxrpc.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/ 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23481 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 37081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23369 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 43345 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20185 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35193 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36569 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33865 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-31T15:40:02.136 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 19177 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_cgroup.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 39513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 62649 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7457 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 13313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 33417 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7913 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10921 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21665 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20857 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 34585 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-31T15:40:02.137 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31513 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq_codel.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 35449 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 49841 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 21969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 66441 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 30217 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10801 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26081 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 36145 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 27401 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-31T15:40:02.138 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10657 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 24769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 48529 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 32809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 26785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 31313 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 28265 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/ 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 932785 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20241 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/ 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/ 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 336969 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-31T15:40:02.139 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 55009 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 1513769 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/ 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 704697 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/ 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 10393 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 867121 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/ 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15929 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/ 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 87737 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 67937 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 98089 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 11073 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-31T15:40:02.140 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 17809 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/ 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 131209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/x25/x25.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/ 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23057 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_algo.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 20545 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 105209 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_user.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/ 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/ 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 15433 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 642383 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 625850 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 7677 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 23498 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.alias.bin 2024-03-31T15:40:02.141 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 9916 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 70732 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 147214 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 233213 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 210 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 72614 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 879 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 257955 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 312782 2024-03-25 03:29 ./lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/ 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/ 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/ 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:drwxr-xr-x root/root 0 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/ 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 169 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/changelog.Debian.gz 2024-03-31T15:40:02.142 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r-- root/root 691 2024-03-25 03:29 ./usr/share/doc/linux-image-6.8.0-g1df0c7e8a235/copyright 2024-03-31T15:40:02.147 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-31T15:40:02.147 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g1df0c7e8a235 2024-03-31T15:40:02.147 DEBUG:teuthology.orchestra.run.smithi090:> egrep '(submenu|menuentry.*6.8.0-g1df0c7e8a235).*{' /boot/grub/grub.cfg 2024-03-31T15:40:02.154 INFO:teuthology.orchestra.run.smithi090.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:40:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:40:02.154 INFO:teuthology.orchestra.run.smithi090.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g1df0c7e8a235 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g1df0c7e8a235-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-31T15:40:02.154 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-31T15:40:02.154 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g1df0c7e8a235 2024-03-31T15:40:02.155 DEBUG:teuthology.orchestra.run.smithi090:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-31T15:40:02.200 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi090.front.sepia.ceph.com to complete... 2024-03-31T15:40:02.800 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub' 2024-03-31T15:40:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-31T15:40:02.822 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-03-31T15:40:03.118 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-6.8.0-g1df0c7e8a235 2024-03-31T15:40:03.125 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-6.8.0-g1df0c7e8a235 2024-03-31T15:40:03.533 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-31T15:40:03.538 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-31T15:40:03.579 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-31T15:40:03.584 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-31T15:40:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-31T15:40:03.700 INFO:teuthology.orchestra.run.smithi090.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-31T15:40:03.701 INFO:teuthology.orchestra.run.smithi090.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-31T15:40:03.719 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-03-31T15:40:27.838 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-31T15:40:27.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-31T15:40:27.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:40:34.897 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-31T15:40:34.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:40:34.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:41:09.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-31T15:41:18.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:41:18.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:41:21.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-31T15:41:27.846 DEBUG:teuthology.orchestra.remote:timed out 2024-03-31T15:41:33.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:41:33.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:41:36.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-31T15:41:36.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:41:37.542 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-31T15:41:38.909 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-31T15:41:38.909 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g1df0c7e8a235"... 2024-03-31T15:41:38.910 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-31T15:41:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:6.8.0-g1df0c7e8a235 2024-03-31T15:41:38.917 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-31T15:41:38.918 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T15:41:38.918 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-31T15:41:39.918 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-31T15:41:39.919 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T15:41:40.239 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-03-31T15:41:40.296 DEBUG:teuthology.parallel:result is None 2024-03-31T15:41:52.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-31T15:42:07.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:42:07.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:42:07.602 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-31T15:42:08.983 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:42:08.983 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g1df0c7e8a235"... 2024-03-31T15:42:08.983 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-31T15:42:08.991 INFO:teuthology.orchestra.run.smithi090.stdout:6.8.0-g1df0c7e8a235 2024-03-31T15:42:08.992 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-31T15:42:08.992 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-31T15:42:08.992 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-31T15:42:09.993 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-31T15:42:09.994 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-31T15:42:10.362 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-03-31T15:42:10.397 DEBUG:teuthology.parallel:result is None 2024-03-31T15:42:10.397 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-31T15:42:10.408 INFO:teuthology.task.internal:Creating test directory... 2024-03-31T15:42:10.409 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T15:42:10.412 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-31T15:42:10.421 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-31T15:42:10.429 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-31T15:42:10.452 INFO:teuthology.task.internal:Creating archive directory... 2024-03-31T15:42:10.453 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T15:42:10.464 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-31T15:42:10.511 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-31T15:42:10.519 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-31T15:42:10.519 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T15:42:10.549 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-31T15:42:10.575 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T15:42:10.588 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T15:42:10.613 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T15:42:10.628 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-31T15:42:10.631 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-31T15:42:10.639 INFO:teuthology.task.internal:Configuring sudo... 2024-03-31T15:42:10.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T15:42:10.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-31T15:42:10.662 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-31T15:42:10.673 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-31T15:42:10.673 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T15:42:10.708 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-31T15:42:10.717 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T15:42:10.766 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T15:42:10.822 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:42:10.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T15:42:10.910 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-31T15:42:10.919 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-31T15:42:10.970 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:42:10.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-31T15:42:11.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-03-31T15:42:11.053 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-31T15:42:11.192 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-31T15:42:11.199 INFO:teuthology.task.internal:Starting timer... 2024-03-31T15:42:11.199 INFO:teuthology.run_tasks:Running task pcp... 2024-03-31T15:42:11.215 INFO:teuthology.run_tasks:Running task selinux... 2024-03-31T15:42:11.223 DEBUG:teuthology.task.selinux:Excluding smithi090: OS 'ubuntu' does not support SELinux 2024-03-31T15:42:11.223 DEBUG:teuthology.task.selinux:Excluding smithi102: OS 'ubuntu' does not support SELinux 2024-03-31T15:42:11.223 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-31T15:42:11.224 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-31T15:42:11.224 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-31T15:42:11.224 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-31T15:42:11.237 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-31T15:42:11.243 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-31T15:42:11.396 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-31T15:42:11.412 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-31T15:42:11.414 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-31T15:49:53.319 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2024-03-31T15:49:53.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:49:53.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:49:53.403 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-31T15:49:53.517 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-31T15:49:53.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-31T15:49:53.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-31T15:49:53.600 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-31T15:49:53.733 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-31T15:49:53.733 INFO:teuthology.run_tasks:Running task clock... 2024-03-31T15:49:53.743 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-31T15:49:53.743 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T15:49:53.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:49:53.747 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-31T15:49:53.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Command line: ntpd -gq 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: ---------------------------------------------------- 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: ntp-4 is maintained by Network Time Foundation, 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-31T15:49:53.782 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: corporation. Support and training for ntp-4 are 2024-03-31T15:49:53.783 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: available at https://www.nwtime.org/support 2024-03-31T15:49:53.783 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: ---------------------------------------------------- 2024-03-31T15:49:53.783 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: proto: precision = 0.042 usec (-24) 2024-03-31T15:49:53.783 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: basedate set to 2022-02-04 2024-03-31T15:49:53.783 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: gps base set to 2022-02-06 (week 2196) 2024-03-31T15:49:53.784 INFO:teuthology.orchestra.run.smithi090.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T15:49:53.784 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T15:49:53.784 INFO:teuthology.orchestra.run.smithi090.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-31T15:49:53.785 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: restrict ::: KOD does nothing without LIMITED. 2024-03-31T15:49:53.785 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen and drop on 0 v6wildcard [::]:123 2024-03-31T15:49:53.785 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-31T15:49:53.786 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen normally on 2 lo 127.0.0.1:123 2024-03-31T15:49:53.786 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen normally on 3 enp3s0f1 172.21.15.90:123 2024-03-31T15:49:53.786 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen normally on 4 lo [::1]:123 2024-03-31T15:49:53.786 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d767%5]:123 2024-03-31T15:49:53.786 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:49:53 ntpd[17218]: Listening on routing socket on fd #22 for interface updates 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Command line: ntpd -gq 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: ---------------------------------------------------- 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: ntp-4 is maintained by Network Time Foundation, 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: corporation. Support and training for ntp-4 are 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: available at https://www.nwtime.org/support 2024-03-31T15:49:53.814 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: ---------------------------------------------------- 2024-03-31T15:49:53.815 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: proto: precision = 0.043 usec (-24) 2024-03-31T15:49:53.815 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: basedate set to 2022-02-04 2024-03-31T15:49:53.815 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: gps base set to 2022-02-06 (week 2196) 2024-03-31T15:49:53.815 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T15:49:53.816 INFO:teuthology.orchestra.run.smithi102.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-31T15:49:53.816 INFO:teuthology.orchestra.run.smithi102.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: restrict ::: KOD does nothing without LIMITED. 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen and drop on 0 v6wildcard [::]:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen normally on 2 lo 127.0.0.1:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen normally on 3 enp3s0f1 172.21.15.102:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen normally on 4 lo [::1]:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d769%5]:123 2024-03-31T15:49:53.817 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:49:53 ntpd[17162]: Listening on routing socket on fd #22 for interface updates 2024-03-31T15:49:54.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:54.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:54.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:54.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:54.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:54.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:54 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:55.782 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:55 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:55 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:55.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:55 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:56 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:56.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:56 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:57.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:57 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:57 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:58.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:58 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:58.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:58.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:58 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:59.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:49:59 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:49:59.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:49:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:49:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:49:59 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:50:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:00.783 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:31 Mar 15:50:00 ntpd[17218]: ntpd: time slew -0.000276 s 2024-03-31T15:50:00.784 INFO:teuthology.orchestra.run.smithi090.stdout:ntpd: time slew -0.000276s 2024-03-31T15:50:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-31T15:50:00.785 INFO:teuthology.orchestra.run.smithi090.stderr:31 Mar 15:50:00 ntpd[17218]: can't open /var/log/ntpstats/loopstats.20240331: Permission denied 2024-03-31T15:50:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:00.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:00.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:00.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:50:00.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:00 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:00.875 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: can't open /var/log/ntpstats/rawstats.20240331: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: can't open /var/log/ntpstats/peerstats.20240331: Permission denied 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:31 Mar 15:50:01 ntpd[17162]: ntpd: time slew +0.000583 s 2024-03-31T15:50:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:ntpd: time slew +0.000583s 2024-03-31T15:50:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-31T15:50:01.816 INFO:teuthology.orchestra.run.smithi102.stderr:31 Mar 15:50:01 ntpd[17162]: can't open /var/log/ntpstats/loopstats.20240331: Permission denied 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================== 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:01.906 INFO:teuthology.orchestra.run.smithi102.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:50:01.907 INFO:teuthology.run_tasks:Running task install... 2024-03-31T15:50:01.915 DEBUG:teuthology.task.install:project ceph 2024-03-31T15:50:01.916 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}} 2024-03-31T15:50:01.916 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'} 2024-03-31T15:50:01.916 INFO:teuthology.task.install:Using flavor: default 2024-03-31T15:50:01.922 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-31T15:50:01.922 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-31T15:50:01.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-key list | grep Ceph 2024-03-31T15:50:01.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-key list | grep Ceph 2024-03-31T15:50:02.026 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-31T15:50:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-31T15:50:02.077 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-31T15:50:02.077 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph.com (release key) 2024-03-31T15:50:02.078 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-31T15:50:02.078 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-31T15:50:02.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:50:02.101 INFO:teuthology.orchestra.run.smithi102.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-31T15:50:02.102 INFO:teuthology.orchestra.run.smithi102.stdout:uid [ unknown] Ceph.com (release key) 2024-03-31T15:50:02.103 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-31T15:50:02.103 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-31T15:50:02.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:50:02.250 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default/ 2024-03-31T15:50:02.250 INFO:teuthology.task.install.deb:Package version is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:50:02.310 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default/ 2024-03-31T15:50:02.310 INFO:teuthology.task.install.deb:Package version is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:50:02.362 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:50:02.362 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-31T15:50:02.415 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get update 2024-03-31T15:50:02.489 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:50:02.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-31T15:50:02.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2024-03-31T15:50:02.606 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-31T15:50:02.618 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-31T15:50:02.620 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-31T15:50:02.641 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-31T15:50:02.706 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-31T15:50:02.706 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-31T15:50:02.717 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-31T15:50:02.719 INFO:teuthology.orchestra.run.smithi102.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy InRelease 2024-03-31T15:50:02.740 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-31T15:50:02.746 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy InRelease 2024-03-31T15:50:02.759 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-31T15:50:02.767 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-31T15:50:02.789 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-31T15:50:02.800 INFO:teuthology.orchestra.run.smithi102.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-31T15:50:02.810 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-31T15:50:02.840 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-31T15:50:02.842 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-03-31T15:50:02.927 INFO:teuthology.orchestra.run.smithi102.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-03-31T15:50:03.191 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 29.2 kB in 1s (52.3 kB/s) 2024-03-31T15:50:03.289 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 29.2 kB in 1s (52.6 kB/s) 2024-03-31T15:50:04.159 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-03-31T15:50:04.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2574-gd2271579-1jammy cephadm=19.0.0-2574-gd2271579-1jammy ceph-mds=19.0.0-2574-gd2271579-1jammy ceph-mgr=19.0.0-2574-gd2271579-1jammy ceph-common=19.0.0-2574-gd2271579-1jammy ceph-fuse=19.0.0-2574-gd2271579-1jammy ceph-test=19.0.0-2574-gd2271579-1jammy ceph-volume=19.0.0-2574-gd2271579-1jammy radosgw=19.0.0-2574-gd2271579-1jammy python3-rados=19.0.0-2574-gd2271579-1jammy python3-rgw=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy python3-rbd=19.0.0-2574-gd2271579-1jammy libcephfs2=19.0.0-2574-gd2271579-1jammy libcephfs-dev=19.0.0-2574-gd2271579-1jammy librados2=19.0.0-2574-gd2271579-1jammy librbd1=19.0.0-2574-gd2271579-1jammy rbd-fuse=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy cephfs-shell=19.0.0-2574-gd2271579-1jammy cephfs-top=19.0.0-2574-gd2271579-1jammy cephfs-mirror=19.0.0-2574-gd2271579-1jammy 2024-03-31T15:50:04.246 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-03-31T15:50:04.248 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-03-31T15:50:04.261 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2574-gd2271579-1jammy cephadm=19.0.0-2574-gd2271579-1jammy ceph-mds=19.0.0-2574-gd2271579-1jammy ceph-mgr=19.0.0-2574-gd2271579-1jammy ceph-common=19.0.0-2574-gd2271579-1jammy ceph-fuse=19.0.0-2574-gd2271579-1jammy ceph-test=19.0.0-2574-gd2271579-1jammy ceph-volume=19.0.0-2574-gd2271579-1jammy radosgw=19.0.0-2574-gd2271579-1jammy python3-rados=19.0.0-2574-gd2271579-1jammy python3-rgw=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy python3-rbd=19.0.0-2574-gd2271579-1jammy libcephfs2=19.0.0-2574-gd2271579-1jammy libcephfs-dev=19.0.0-2574-gd2271579-1jammy librados2=19.0.0-2574-gd2271579-1jammy librbd1=19.0.0-2574-gd2271579-1jammy rbd-fuse=19.0.0-2574-gd2271579-1jammy python3-cephfs=19.0.0-2574-gd2271579-1jammy cephfs-shell=19.0.0-2574-gd2271579-1jammy cephfs-top=19.0.0-2574-gd2271579-1jammy cephfs-mirror=19.0.0-2574-gd2271579-1jammy 2024-03-31T15:50:04.334 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-03-31T15:50:04.420 INFO:teuthology.orchestra.run.smithi102.stdout:Building dependency tree... 2024-03-31T15:50:04.421 INFO:teuthology.orchestra.run.smithi102.stdout:Reading state information... 2024-03-31T15:50:04.506 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2024-03-31T15:50:04.506 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2024-03-31T15:50:04.594 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:50:04.594 INFO:teuthology.orchestra.run.smithi102.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:50:04.594 INFO:teuthology.orchestra.run.smithi102.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:50:04.594 INFO:teuthology.orchestra.run.smithi102.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:50:04.594 INFO:teuthology.orchestra.run.smithi102.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:50:04.595 INFO:teuthology.orchestra.run.smithi102.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:50:04.595 INFO:teuthology.orchestra.run.smithi102.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:50:04.595 INFO:teuthology.orchestra.run.smithi102.stdout:The following additional packages will be installed: 2024-03-31T15:50:04.595 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-31T15:50:04.595 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-31T15:50:04.596 INFO:teuthology.orchestra.run.smithi102.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T15:50:04.596 INFO:teuthology.orchestra.run.smithi102.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-31T15:50:04.597 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-31T15:50:04.598 INFO:teuthology.orchestra.run.smithi102.stdout:Suggested packages: 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout:Recommended packages: 2024-03-31T15:50:04.599 INFO:teuthology.orchestra.run.smithi102.stdout: btrfs-tools 2024-03-31T15:50:04.678 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:50:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:50:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:50:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:50:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T15:50:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-31T15:50:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-31T15:50:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-31T15:50:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-31T15:50:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-31T15:50:04.684 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2024-03-31T15:50:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: btrfs-tools 2024-03-31T15:50:04.746 INFO:teuthology.orchestra.run.smithi102.stdout:The following NEW packages will be installed: 2024-03-31T15:50:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-31T15:50:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-31T15:50:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-31T15:50:04.746 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-31T15:50:04.747 INFO:teuthology.orchestra.run.smithi102.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T15:50:04.747 INFO:teuthology.orchestra.run.smithi102.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-31T15:50:04.747 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-31T15:50:04.747 INFO:teuthology.orchestra.run.smithi102.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-31T15:50:04.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-31T15:50:04.749 INFO:teuthology.orchestra.run.smithi102.stdout: socat unzip xmlstarlet zip 2024-03-31T15:50:04.827 INFO:teuthology.orchestra.run.smithi102.stdout:0 upgraded, 124 newly installed, 0 to remove and 301 not upgraded. 2024-03-31T15:50:04.827 INFO:teuthology.orchestra.run.smithi102.stdout:Need to get 179 MB of archives. 2024-03-31T15:50:04.827 INFO:teuthology.orchestra.run.smithi102.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-31T15:50:04.827 INFO:teuthology.orchestra.run.smithi102.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-31T15:50:04.832 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2024-03-31T15:50:04.832 INFO:teuthology.orchestra.run.smithi090.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-31T15:50:04.832 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-31T15:50:04.832 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-31T15:50:04.832 INFO:teuthology.orchestra.run.smithi090.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-31T15:50:04.833 INFO:teuthology.orchestra.run.smithi090.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-31T15:50:04.834 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-31T15:50:04.835 INFO:teuthology.orchestra.run.smithi090.stdout: socat unzip xmlstarlet zip 2024-03-31T15:50:04.850 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-31T15:50:04.863 INFO:teuthology.orchestra.run.smithi102.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-31T15:50:04.895 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2574-gd2271579-1jammy [3,553 kB] 2024-03-31T15:50:04.912 INFO:teuthology.orchestra.run.smithi090.stdout:0 upgraded, 124 newly installed, 0 to remove and 301 not upgraded. 2024-03-31T15:50:04.913 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 179 MB of archives. 2024-03-31T15:50:04.913 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 821 MB of additional disk space will be used. 2024-03-31T15:50:04.913 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-31T15:50:04.920 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-31T15:50:04.937 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-31T15:50:04.949 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-31T15:50:04.951 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-31T15:50:04.953 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-31T15:50:04.969 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-31T15:50:04.982 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2574-gd2271579-1jammy [3,553 kB] 2024-03-31T15:50:05.020 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-31T15:50:05.033 INFO:teuthology.orchestra.run.smithi102.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-31T15:50:05.035 INFO:teuthology.orchestra.run.smithi102.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-31T15:50:05.049 INFO:teuthology.orchestra.run.smithi102.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-31T15:50:05.053 INFO:teuthology.orchestra.run.smithi102.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-31T15:50:05.053 INFO:teuthology.orchestra.run.smithi102.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-31T15:50:05.064 INFO:teuthology.orchestra.run.smithi102.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-31T15:50:05.064 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-31T15:50:05.066 INFO:teuthology.orchestra.run.smithi102.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-31T15:50:05.068 INFO:teuthology.orchestra.run.smithi102.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-31T15:50:05.070 INFO:teuthology.orchestra.run.smithi102.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-31T15:50:05.070 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-31T15:50:05.073 INFO:teuthology.orchestra.run.smithi102.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-31T15:50:05.085 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-31T15:50:05.096 INFO:teuthology.orchestra.run.smithi102.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-31T15:50:05.098 INFO:teuthology.orchestra.run.smithi102.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-31T15:50:05.100 INFO:teuthology.orchestra.run.smithi102.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-31T15:50:05.100 INFO:teuthology.orchestra.run.smithi102.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-31T15:50:05.101 INFO:teuthology.orchestra.run.smithi102.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-31T15:50:05.102 INFO:teuthology.orchestra.run.smithi102.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-31T15:50:05.102 INFO:teuthology.orchestra.run.smithi102.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-31T15:50:05.103 INFO:teuthology.orchestra.run.smithi102.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-31T15:50:05.103 INFO:teuthology.orchestra.run.smithi102.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-31T15:50:05.129 INFO:teuthology.orchestra.run.smithi102.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-31T15:50:05.130 INFO:teuthology.orchestra.run.smithi102.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-31T15:50:05.130 INFO:teuthology.orchestra.run.smithi102.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-31T15:50:05.132 INFO:teuthology.orchestra.run.smithi102.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-31T15:50:05.133 INFO:teuthology.orchestra.run.smithi102.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-31T15:50:05.148 INFO:teuthology.orchestra.run.smithi102.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2574-gd2271579-1jammy [3,224 kB] 2024-03-31T15:50:05.159 INFO:teuthology.orchestra.run.smithi102.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-31T15:50:05.160 INFO:teuthology.orchestra.run.smithi102.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-31T15:50:05.161 INFO:teuthology.orchestra.run.smithi102.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-31T15:50:05.161 INFO:teuthology.orchestra.run.smithi102.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-31T15:50:05.161 INFO:teuthology.orchestra.run.smithi102.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-31T15:50:05.164 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-31T15:50:05.168 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-31T15:50:05.183 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-31T15:50:05.186 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-31T15:50:05.187 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-31T15:50:05.196 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-31T15:50:05.198 INFO:teuthology.orchestra.run.smithi102.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-31T15:50:05.198 INFO:teuthology.orchestra.run.smithi102.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-31T15:50:05.199 INFO:teuthology.orchestra.run.smithi102.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-31T15:50:05.199 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-31T15:50:05.204 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-31T15:50:05.206 INFO:teuthology.orchestra.run.smithi102.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-31T15:50:05.207 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-31T15:50:05.207 INFO:teuthology.orchestra.run.smithi102.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-31T15:50:05.208 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-31T15:50:05.222 INFO:teuthology.orchestra.run.smithi102.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-31T15:50:05.223 INFO:teuthology.orchestra.run.smithi102.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-31T15:50:05.224 INFO:teuthology.orchestra.run.smithi102.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-31T15:50:05.226 INFO:teuthology.orchestra.run.smithi102.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-31T15:50:05.226 INFO:teuthology.orchestra.run.smithi102.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-31T15:50:05.234 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-31T15:50:05.236 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-31T15:50:05.238 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-31T15:50:05.239 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-31T15:50:05.239 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-31T15:50:05.240 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-31T15:50:05.241 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-31T15:50:05.241 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-31T15:50:05.242 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-31T15:50:05.251 INFO:teuthology.orchestra.run.smithi102.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-31T15:50:05.251 INFO:teuthology.orchestra.run.smithi102.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-31T15:50:05.254 INFO:teuthology.orchestra.run.smithi102.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-31T15:50:05.254 INFO:teuthology.orchestra.run.smithi102.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-31T15:50:05.255 INFO:teuthology.orchestra.run.smithi102.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-31T15:50:05.267 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-31T15:50:05.267 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-31T15:50:05.268 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-31T15:50:05.275 INFO:teuthology.orchestra.run.smithi102.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-31T15:50:05.311 INFO:teuthology.orchestra.run.smithi102.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2574-gd2271579-1jammy [712 kB] 2024-03-31T15:50:05.316 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-31T15:50:05.316 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-31T15:50:05.317 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-31T15:50:05.317 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-31T15:50:05.318 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-31T15:50:05.319 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-31T15:50:05.319 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-31T15:50:05.322 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2574-gd2271579-1jammy [3,224 kB] 2024-03-31T15:50:05.349 INFO:teuthology.orchestra.run.smithi102.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-31T15:50:05.349 INFO:teuthology.orchestra.run.smithi102.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-31T15:50:05.353 INFO:teuthology.orchestra.run.smithi102.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-31T15:50:05.354 INFO:teuthology.orchestra.run.smithi102.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-31T15:50:05.355 INFO:teuthology.orchestra.run.smithi102.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2574-gd2271579-1jammy [356 kB] 2024-03-31T15:50:05.355 INFO:teuthology.orchestra.run.smithi102.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-31T15:50:05.360 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-31T15:50:05.361 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-31T15:50:05.362 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-31T15:50:05.366 INFO:teuthology.orchestra.run.smithi102.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2574-gd2271579-1jammy [32.8 kB] 2024-03-31T15:50:05.367 INFO:teuthology.orchestra.run.smithi102.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2574-gd2271579-1jammy [180 kB] 2024-03-31T15:50:05.387 INFO:teuthology.orchestra.run.smithi102.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-31T15:50:05.389 INFO:teuthology.orchestra.run.smithi102.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-31T15:50:05.390 INFO:teuthology.orchestra.run.smithi102.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-31T15:50:05.395 INFO:teuthology.orchestra.run.smithi102.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2574-gd2271579-1jammy [66.7 kB] 2024-03-31T15:50:05.397 INFO:teuthology.orchestra.run.smithi102.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2574-gd2271579-1jammy [330 kB] 2024-03-31T15:50:05.408 INFO:teuthology.orchestra.run.smithi102.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2574-gd2271579-1jammy [6,660 kB] 2024-03-31T15:50:05.409 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-31T15:50:05.409 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-31T15:50:05.410 INFO:teuthology.orchestra.run.smithi090.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-31T15:50:05.411 INFO:teuthology.orchestra.run.smithi090.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-31T15:50:05.412 INFO:teuthology.orchestra.run.smithi090.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-31T15:50:05.413 INFO:teuthology.orchestra.run.smithi102.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-31T15:50:05.413 INFO:teuthology.orchestra.run.smithi102.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-31T15:50:05.414 INFO:teuthology.orchestra.run.smithi090.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-31T15:50:05.415 INFO:teuthology.orchestra.run.smithi090.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-31T15:50:05.428 INFO:teuthology.orchestra.run.smithi102.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-31T15:50:05.429 INFO:teuthology.orchestra.run.smithi102.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-31T15:50:05.429 INFO:teuthology.orchestra.run.smithi102.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-31T15:50:05.430 INFO:teuthology.orchestra.run.smithi102.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-31T15:50:05.430 INFO:teuthology.orchestra.run.smithi102.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-31T15:50:05.431 INFO:teuthology.orchestra.run.smithi102.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-31T15:50:05.435 INFO:teuthology.orchestra.run.smithi102.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-31T15:50:05.439 INFO:teuthology.orchestra.run.smithi090.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-31T15:50:05.440 INFO:teuthology.orchestra.run.smithi090.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-31T15:50:05.442 INFO:teuthology.orchestra.run.smithi090.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-31T15:50:05.455 INFO:teuthology.orchestra.run.smithi102.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-31T15:50:05.455 INFO:teuthology.orchestra.run.smithi102.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-31T15:50:05.456 INFO:teuthology.orchestra.run.smithi102.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-31T15:50:05.467 INFO:teuthology.orchestra.run.smithi090.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-31T15:50:05.467 INFO:teuthology.orchestra.run.smithi090.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-31T15:50:05.470 INFO:teuthology.orchestra.run.smithi090.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-31T15:50:05.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-31T15:50:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-31T15:50:05.503 INFO:teuthology.orchestra.run.smithi102.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-31T15:50:05.504 INFO:teuthology.orchestra.run.smithi102.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-31T15:50:05.505 INFO:teuthology.orchestra.run.smithi102.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-31T15:50:05.514 INFO:teuthology.orchestra.run.smithi102.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-31T15:50:05.515 INFO:teuthology.orchestra.run.smithi102.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-31T15:50:05.548 INFO:teuthology.orchestra.run.smithi090.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2574-gd2271579-1jammy [712 kB] 2024-03-31T15:50:05.548 INFO:teuthology.orchestra.run.smithi090.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-31T15:50:05.549 INFO:teuthology.orchestra.run.smithi102.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-31T15:50:05.549 INFO:teuthology.orchestra.run.smithi102.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-31T15:50:05.549 INFO:teuthology.orchestra.run.smithi090.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-31T15:50:05.550 INFO:teuthology.orchestra.run.smithi102.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-31T15:50:05.551 INFO:teuthology.orchestra.run.smithi090.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-31T15:50:05.552 INFO:teuthology.orchestra.run.smithi090.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-31T15:50:05.553 INFO:teuthology.orchestra.run.smithi090.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-31T15:50:05.582 INFO:teuthology.orchestra.run.smithi090.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-31T15:50:05.584 INFO:teuthology.orchestra.run.smithi090.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-31T15:50:05.585 INFO:teuthology.orchestra.run.smithi090.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-31T15:50:05.589 INFO:teuthology.orchestra.run.smithi102.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-31T15:50:05.590 INFO:teuthology.orchestra.run.smithi102.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-31T15:50:05.591 INFO:teuthology.orchestra.run.smithi102.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-31T15:50:05.600 INFO:teuthology.orchestra.run.smithi102.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-31T15:50:05.600 INFO:teuthology.orchestra.run.smithi102.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-31T15:50:05.601 INFO:teuthology.orchestra.run.smithi102.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-31T15:50:05.602 INFO:teuthology.orchestra.run.smithi090.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2574-gd2271579-1jammy [356 kB] 2024-03-31T15:50:05.605 INFO:teuthology.orchestra.run.smithi102.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-31T15:50:05.614 INFO:teuthology.orchestra.run.smithi090.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2574-gd2271579-1jammy [32.8 kB] 2024-03-31T15:50:05.615 INFO:teuthology.orchestra.run.smithi090.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2574-gd2271579-1jammy [180 kB] 2024-03-31T15:50:05.619 INFO:teuthology.orchestra.run.smithi090.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-31T15:50:05.623 INFO:teuthology.orchestra.run.smithi090.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-31T15:50:05.624 INFO:teuthology.orchestra.run.smithi090.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-31T15:50:05.625 INFO:teuthology.orchestra.run.smithi090.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-31T15:50:05.625 INFO:teuthology.orchestra.run.smithi090.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-31T15:50:05.626 INFO:teuthology.orchestra.run.smithi090.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-31T15:50:05.627 INFO:teuthology.orchestra.run.smithi102.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-31T15:50:05.653 INFO:teuthology.orchestra.run.smithi090.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2574-gd2271579-1jammy [66.7 kB] 2024-03-31T15:50:05.655 INFO:teuthology.orchestra.run.smithi090.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2574-gd2271579-1jammy [330 kB] 2024-03-31T15:50:05.658 INFO:teuthology.orchestra.run.smithi090.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-31T15:50:05.658 INFO:teuthology.orchestra.run.smithi090.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-31T15:50:05.662 INFO:teuthology.orchestra.run.smithi090.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-31T15:50:05.664 INFO:teuthology.orchestra.run.smithi090.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-31T15:50:05.666 INFO:teuthology.orchestra.run.smithi090.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-31T15:50:05.668 INFO:teuthology.orchestra.run.smithi090.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2574-gd2271579-1jammy [6,660 kB] 2024-03-31T15:50:05.681 INFO:teuthology.orchestra.run.smithi090.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-31T15:50:05.686 INFO:teuthology.orchestra.run.smithi090.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-31T15:50:05.689 INFO:teuthology.orchestra.run.smithi090.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-31T15:50:05.690 INFO:teuthology.orchestra.run.smithi090.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-31T15:50:05.691 INFO:teuthology.orchestra.run.smithi090.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-31T15:50:05.705 INFO:teuthology.orchestra.run.smithi090.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-31T15:50:05.711 INFO:teuthology.orchestra.run.smithi090.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-31T15:50:05.712 INFO:teuthology.orchestra.run.smithi090.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-31T15:50:05.716 INFO:teuthology.orchestra.run.smithi090.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-31T15:50:05.716 INFO:teuthology.orchestra.run.smithi090.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-31T15:50:05.729 INFO:teuthology.orchestra.run.smithi090.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-31T15:50:05.730 INFO:teuthology.orchestra.run.smithi090.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-31T15:50:05.731 INFO:teuthology.orchestra.run.smithi090.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-31T15:50:05.732 INFO:teuthology.orchestra.run.smithi090.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-31T15:50:05.742 INFO:teuthology.orchestra.run.smithi090.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-31T15:50:05.754 INFO:teuthology.orchestra.run.smithi090.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-31T15:50:05.755 INFO:teuthology.orchestra.run.smithi090.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-31T15:50:05.757 INFO:teuthology.orchestra.run.smithi090.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-31T15:50:05.757 INFO:teuthology.orchestra.run.smithi090.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-31T15:50:05.820 INFO:teuthology.orchestra.run.smithi102.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2574-gd2271579-1jammy [112 kB] 2024-03-31T15:50:05.821 INFO:teuthology.orchestra.run.smithi102.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2574-gd2271579-1jammy [466 kB] 2024-03-31T15:50:05.836 INFO:teuthology.orchestra.run.smithi102.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2574-gd2271579-1jammy [25.3 MB] 2024-03-31T15:50:06.186 INFO:teuthology.orchestra.run.smithi090.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2574-gd2271579-1jammy [112 kB] 2024-03-31T15:50:06.187 INFO:teuthology.orchestra.run.smithi090.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2574-gd2271579-1jammy [466 kB] 2024-03-31T15:50:06.191 INFO:teuthology.orchestra.run.smithi090.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2574-gd2271579-1jammy [25.3 MB] 2024-03-31T15:50:07.354 INFO:teuthology.orchestra.run.smithi102.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2574-gd2271579-1jammy [5,086 kB] 2024-03-31T15:50:07.668 INFO:teuthology.orchestra.run.smithi102.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2574-gd2271579-1jammy [244 kB] 2024-03-31T15:50:07.673 INFO:teuthology.orchestra.run.smithi102.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2574-gd2271579-1jammy [124 kB] 2024-03-31T15:50:07.677 INFO:teuthology.orchestra.run.smithi102.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2574-gd2271579-1jammy [1,526 kB] 2024-03-31T15:50:07.759 INFO:teuthology.orchestra.run.smithi102.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2574-gd2271579-1jammy [6,259 kB] 2024-03-31T15:50:08.065 INFO:teuthology.orchestra.run.smithi090.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2574-gd2271579-1jammy [5,086 kB] 2024-03-31T15:50:08.152 INFO:teuthology.orchestra.run.smithi102.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2574-gd2271579-1jammy [22.8 MB] 2024-03-31T15:50:08.427 INFO:teuthology.orchestra.run.smithi090.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2574-gd2271579-1jammy [244 kB] 2024-03-31T15:50:08.433 INFO:teuthology.orchestra.run.smithi090.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2574-gd2271579-1jammy [124 kB] 2024-03-31T15:50:08.437 INFO:teuthology.orchestra.run.smithi090.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2574-gd2271579-1jammy [1,526 kB] 2024-03-31T15:50:08.566 INFO:teuthology.orchestra.run.smithi090.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2574-gd2271579-1jammy [6,259 kB] 2024-03-31T15:50:09.025 INFO:teuthology.orchestra.run.smithi090.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2574-gd2271579-1jammy [22.8 MB] 2024-03-31T15:50:09.523 INFO:teuthology.orchestra.run.smithi102.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2574-gd2271579-1jammy [14.0 kB] 2024-03-31T15:50:09.523 INFO:teuthology.orchestra.run.smithi102.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2574-gd2271579-1jammy [870 kB] 2024-03-31T15:50:09.546 INFO:teuthology.orchestra.run.smithi102.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2574-gd2271579-1jammy [2,435 kB] 2024-03-31T15:50:09.707 INFO:teuthology.orchestra.run.smithi102.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2574-gd2271579-1jammy [791 kB] 2024-03-31T15:50:09.760 INFO:teuthology.orchestra.run.smithi102.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2574-gd2271579-1jammy [147 kB] 2024-03-31T15:50:09.765 INFO:teuthology.orchestra.run.smithi102.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2574-gd2271579-1jammy [2,313 kB] 2024-03-31T15:50:09.909 INFO:teuthology.orchestra.run.smithi102.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2574-gd2271579-1jammy [8,625 kB] 2024-03-31T15:50:10.429 INFO:teuthology.orchestra.run.smithi102.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2574-gd2271579-1jammy [14.2 kB] 2024-03-31T15:50:10.429 INFO:teuthology.orchestra.run.smithi102.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2574-gd2271579-1jammy [50.3 MB] 2024-03-31T15:50:10.654 INFO:teuthology.orchestra.run.smithi090.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2574-gd2271579-1jammy [14.0 kB] 2024-03-31T15:50:10.655 INFO:teuthology.orchestra.run.smithi090.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2574-gd2271579-1jammy [870 kB] 2024-03-31T15:50:10.716 INFO:teuthology.orchestra.run.smithi090.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2574-gd2271579-1jammy [2,435 kB] 2024-03-31T15:50:10.894 INFO:teuthology.orchestra.run.smithi090.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2574-gd2271579-1jammy [791 kB] 2024-03-31T15:50:10.952 INFO:teuthology.orchestra.run.smithi090.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2574-gd2271579-1jammy [147 kB] 2024-03-31T15:50:10.957 INFO:teuthology.orchestra.run.smithi090.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2574-gd2271579-1jammy [2,313 kB] 2024-03-31T15:50:11.131 INFO:teuthology.orchestra.run.smithi090.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2574-gd2271579-1jammy [8,625 kB] 2024-03-31T15:50:11.771 INFO:teuthology.orchestra.run.smithi090.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2574-gd2271579-1jammy [14.2 kB] 2024-03-31T15:50:11.772 INFO:teuthology.orchestra.run.smithi090.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2574-gd2271579-1jammy [50.3 MB] 2024-03-31T15:50:13.459 INFO:teuthology.orchestra.run.smithi102.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2574-gd2271579-1jammy [121 kB] 2024-03-31T15:50:13.460 INFO:teuthology.orchestra.run.smithi102.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2574-gd2271579-1jammy [232 kB] 2024-03-31T15:50:13.468 INFO:teuthology.orchestra.run.smithi102.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2574-gd2271579-1jammy [34.0 kB] 2024-03-31T15:50:13.474 INFO:teuthology.orchestra.run.smithi102.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2574-gd2271579-1jammy [23.4 kB] 2024-03-31T15:50:13.475 INFO:teuthology.orchestra.run.smithi102.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2574-gd2271579-1jammy [38.9 kB] 2024-03-31T15:50:13.478 INFO:teuthology.orchestra.run.smithi102.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2574-gd2271579-1jammy [13.2 MB] 2024-03-31T15:50:14.269 INFO:teuthology.orchestra.run.smithi102.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2574-gd2271579-1jammy [91.2 kB] 2024-03-31T15:50:14.807 INFO:teuthology.orchestra.run.smithi090.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2574-gd2271579-1jammy [121 kB] 2024-03-31T15:50:14.808 INFO:teuthology.orchestra.run.smithi090.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2574-gd2271579-1jammy [232 kB] 2024-03-31T15:50:14.810 INFO:teuthology.orchestra.run.smithi090.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2574-gd2271579-1jammy [34.0 kB] 2024-03-31T15:50:14.811 INFO:teuthology.orchestra.run.smithi090.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2574-gd2271579-1jammy [23.4 kB] 2024-03-31T15:50:14.811 INFO:teuthology.orchestra.run.smithi090.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2574-gd2271579-1jammy [38.9 kB] 2024-03-31T15:50:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2574-gd2271579-1jammy [13.2 MB] 2024-03-31T15:50:14.900 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 179 MB in 10s (18.8 MB/s) 2024-03-31T15:50:15.049 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-31T15:50:15.114 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-31T15:50:15.117 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-31T15:50:15.216 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-31T15:50:15.240 INFO:teuthology.orchestra.run.smithi090.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240330.170739/d2271579d5e735cfd6d436b8349c004f06527cb0/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2574-gd2271579-1jammy [91.2 kB] 2024-03-31T15:50:15.584 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-31T15:50:15.597 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-31T15:50:15.634 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:50:15.904 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 179 MB in 10s (17.2 MB/s) 2024-03-31T15:50:15.978 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cmd2. 2024-03-31T15:50:15.991 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-31T15:50:16.028 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-31T15:50:16.194 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-31T15:50:16.262 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-31T15:50:16.265 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-31T15:50:16.350 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-31T15:50:16.373 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-31T15:50:16.386 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-31T15:50:16.473 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:50:16.670 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-31T15:50:16.684 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-31T15:50:16.729 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:50:16.852 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-31T15:50:16.865 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-31T15:50:16.910 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:50:17.040 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cmd2. 2024-03-31T15:50:17.054 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-31T15:50:17.090 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-31T15:50:17.254 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-31T15:50:17.269 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-31T15:50:17.305 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:50:17.402 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-31T15:50:17.416 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-31T15:50:17.494 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:50:17.716 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-31T15:50:17.731 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:17.783 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:17.841 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-31T15:50:17.856 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-31T15:50:17.899 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:50:18.178 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-31T15:50:18.192 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:18.228 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-31T15:50:18.236 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:18.243 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-31T15:50:18.278 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:50:18.589 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-31T15:50:18.604 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:18.639 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:18.716 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-31T15:50:18.731 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:18.783 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:18.959 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-31T15:50:18.973 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-31T15:50:19.042 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:50:19.153 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-31T15:50:19.168 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:19.203 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:19.362 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librados2. 2024-03-31T15:50:19.377 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../010-librados2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:19.420 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:19.575 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-31T15:50:19.590 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-31T15:50:19.633 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:50:19.995 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-31T15:50:20.010 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-31T15:50:20.053 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:50:20.065 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librbd1. 2024-03-31T15:50:20.079 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../011-librbd1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:20.123 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:20.357 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librados2. 2024-03-31T15:50:20.371 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../010-librados2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:20.406 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:20.501 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libcephfs2. 2024-03-31T15:50:20.515 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:20.551 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:20.812 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rados. 2024-03-31T15:50:20.826 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:20.852 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librbd1. 2024-03-31T15:50:20.867 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../011-librbd1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:20.895 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:20.944 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:21.214 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-31T15:50:21.229 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:21.273 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:21.364 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs2. 2024-03-31T15:50:21.377 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:21.500 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:21.542 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cephfs. 2024-03-31T15:50:21.556 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:21.609 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:21.810 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rados. 2024-03-31T15:50:21.823 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:21.860 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:21.861 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-31T15:50:21.875 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:22.180 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-31T15:50:22.194 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:22.206 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-prettytable. 2024-03-31T15:50:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-31T15:50:22.238 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:22.264 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-31T15:50:22.499 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cephfs. 2024-03-31T15:50:22.513 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:22.525 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rbd. 2024-03-31T15:50:22.539 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:22.549 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:22.583 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:22.862 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-31T15:50:22.876 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:22.920 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:22.953 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-31T15:50:22.967 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-31T15:50:23.011 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:50:23.232 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-prettytable. 2024-03-31T15:50:23.247 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-31T15:50:23.289 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-31T15:50:23.290 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-31T15:50:23.304 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-31T15:50:23.355 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:50:23.535 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rbd. 2024-03-31T15:50:23.550 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:23.593 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-31T15:50:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-31T15:50:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:50:23.956 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-31T15:50:23.971 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-31T15:50:24.003 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua5.1. 2024-03-31T15:50:24.006 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:50:24.018 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-31T15:50:24.062 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:50:24.259 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-31T15:50:24.274 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-31T15:50:24.317 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:50:24.365 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-any. 2024-03-31T15:50:24.379 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-31T15:50:24.423 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-31T15:50:24.642 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-31T15:50:24.654 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-31T15:50:24.697 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:50:24.700 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package zip. 2024-03-31T15:50:24.715 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-31T15:50:24.767 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking zip (3.0-12build2) ... 2024-03-31T15:50:25.034 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package lua5.1. 2024-03-31T15:50:25.049 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-31T15:50:25.054 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package unzip. 2024-03-31T15:50:25.068 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-31T15:50:25.084 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:50:25.103 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:50:25.430 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package lua-any. 2024-03-31T15:50:25.445 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-31T15:50:25.489 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-31T15:50:25.507 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package luarocks. 2024-03-31T15:50:25.522 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-31T15:50:25.566 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:50:25.742 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package zip. 2024-03-31T15:50:25.757 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-31T15:50:25.792 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking zip (3.0-12build2) ... 2024-03-31T15:50:25.957 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librgw2. 2024-03-31T15:50:25.971 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../027-librgw2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:26.015 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:26.113 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package unzip. 2024-03-31T15:50:26.127 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-31T15:50:26.171 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:50:26.526 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rgw. 2024-03-31T15:50:26.539 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:26.568 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package luarocks. 2024-03-31T15:50:26.583 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-31T15:50:26.586 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:26.676 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:50:26.906 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-31T15:50:26.920 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-31T15:50:26.964 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:50:27.061 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librgw2. 2024-03-31T15:50:27.076 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../027-librgw2_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:27.120 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:27.242 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libradosstriper1. 2024-03-31T15:50:27.256 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:27.291 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:27.632 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rgw. 2024-03-31T15:50:27.645 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:27.653 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-common. 2024-03-31T15:50:27.668 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:27.689 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:27.706 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:28.019 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-31T15:50:28.034 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-31T15:50:28.127 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:50:28.430 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libradosstriper1. 2024-03-31T15:50:28.445 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:28.489 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:28.792 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-base. 2024-03-31T15:50:28.806 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:28.851 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:28.876 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-common. 2024-03-31T15:50:28.891 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:28.926 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:29.421 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-31T15:50:29.434 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-31T15:50:29.479 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:50:29.807 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cheroot. 2024-03-31T15:50:29.821 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-31T15:50:29.865 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:50:30.185 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-31T15:50:30.199 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-31T15:50:30.208 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-base. 2024-03-31T15:50:30.224 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:30.260 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:50:30.275 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:30.738 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-31T15:50:30.752 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-31T15:50:30.788 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:50:30.873 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-31T15:50:30.887 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-31T15:50:30.931 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:50:31.066 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-31T15:50:31.080 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-31T15:50:31.124 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:50:31.259 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cheroot. 2024-03-31T15:50:31.274 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-31T15:50:31.317 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:50:31.427 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-tempora. 2024-03-31T15:50:31.441 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-31T15:50:31.510 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-31T15:50:31.606 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-31T15:50:31.621 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-31T15:50:31.794 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:50:31.813 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-portend. 2024-03-31T15:50:31.827 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-31T15:50:31.871 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-31T15:50:32.202 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-31T15:50:32.214 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-31T15:50:32.217 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-31T15:50:32.232 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-31T15:50:32.249 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-31T15:50:32.275 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:50:32.535 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-31T15:50:32.550 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-31T15:50:32.601 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:50:32.612 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-31T15:50:32.627 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-31T15:50:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:50:32.947 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-natsort. 2024-03-31T15:50:32.950 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempora. 2024-03-31T15:50:32.962 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-31T15:50:32.964 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-31T15:50:32.997 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-31T15:50:33.000 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-31T15:50:33.250 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-logutils. 2024-03-31T15:50:33.264 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-31T15:50:33.295 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-portend. 2024-03-31T15:50:33.310 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-31T15:50:33.333 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-31T15:50:33.353 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-31T15:50:33.636 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-31T15:50:33.650 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-31T15:50:33.650 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-31T15:50:33.664 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-31T15:50:33.694 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:50:33.708 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-31T15:50:34.028 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-31T15:50:34.030 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-mako. 2024-03-31T15:50:34.042 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-31T15:50:34.045 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-31T15:50:34.078 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:50:34.088 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:50:34.407 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-natsort. 2024-03-31T15:50:34.417 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-31T15:50:34.421 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-31T15:50:34.431 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-31T15:50:34.466 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-31T15:50:34.467 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:50:34.728 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-31T15:50:34.742 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-31T15:50:34.762 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-logutils. 2024-03-31T15:50:34.776 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-31T15:50:34.786 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:50:34.812 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-31T15:50:35.107 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-31T15:50:35.121 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-31T15:50:35.132 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-webob. 2024-03-31T15:50:35.145 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-31T15:50:35.165 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:50:35.189 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:50:35.503 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-mako. 2024-03-31T15:50:35.517 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-31T15:50:35.525 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-waitress. 2024-03-31T15:50:35.540 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-31T15:50:35.561 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:50:35.587 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:50:35.849 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-31T15:50:35.863 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-31T15:50:35.898 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:50:35.920 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-tempita. 2024-03-31T15:50:35.934 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-31T15:50:35.978 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:50:36.178 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-31T15:50:36.192 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-31T15:50:36.236 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:50:36.248 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-paste. 2024-03-31T15:50:36.262 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-31T15:50:36.298 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:50:36.548 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webob. 2024-03-31T15:50:36.562 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-31T15:50:36.606 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:50:36.634 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-31T15:50:36.649 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-31T15:50:36.692 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:50:36.974 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-31T15:50:36.988 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-31T15:50:37.028 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:50:37.307 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-webtest. 2024-03-31T15:50:37.321 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-31T15:50:37.365 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-31T15:50:37.852 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-waitress. 2024-03-31T15:50:37.869 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-31T15:50:37.910 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pecan. 2024-03-31T15:50:37.921 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:50:37.924 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-31T15:50:38.231 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempita. 2024-03-31T15:50:38.246 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-31T15:50:38.290 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:50:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:50:38.610 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-paste. 2024-03-31T15:50:38.625 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-31T15:50:38.668 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:50:38.773 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-certifi. 2024-03-31T15:50:38.787 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-31T15:50:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-31T15:50:38.981 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-31T15:50:38.996 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-31T15:50:39.039 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:50:39.176 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-idna. 2024-03-31T15:50:39.190 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-31T15:50:39.225 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-31T15:50:39.334 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-31T15:50:39.349 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-31T15:50:39.393 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:50:39.461 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-urllib3. 2024-03-31T15:50:39.476 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-31T15:50:39.511 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:50:39.704 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webtest. 2024-03-31T15:50:39.719 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-31T15:50:39.762 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-31T15:50:39.823 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-requests. 2024-03-31T15:50:39.838 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-31T15:50:39.881 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:50:40.092 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pecan. 2024-03-31T15:50:40.108 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-31T15:50:40.126 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-31T15:50:40.140 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-31T15:50:40.150 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:50:40.175 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:50:40.487 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-31T15:50:40.502 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:40.591 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-certifi. 2024-03-31T15:50:40.606 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-31T15:50:40.649 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-31T15:50:41.036 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-idna. 2024-03-31T15:50:41.051 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-31T15:50:41.075 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-31T15:50:41.090 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:41.102 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-31T15:50:41.125 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:41.403 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-31T15:50:41.417 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-31T15:50:41.494 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:50:41.522 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-urllib3. 2024-03-31T15:50:41.538 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-31T15:50:41.589 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:50:41.806 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-openssl. 2024-03-31T15:50:41.820 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-31T15:50:41.864 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-31T15:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests. 2024-03-31T15:50:41.992 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-31T15:50:42.044 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:50:42.209 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr. 2024-03-31T15:50:42.224 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:42.292 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:42.614 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-31T15:50:42.620 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mon. 2024-03-31T15:50:42.630 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-31T15:50:42.635 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:42.670 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:42.672 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:50:43.019 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-31T15:50:43.027 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:43.069 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:43.207 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-osd. 2024-03-31T15:50:43.221 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:43.257 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:43.458 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-31T15:50:43.470 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:43.516 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:43.812 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-31T15:50:43.824 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-31T15:50:43.862 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:50:44.035 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph. 2024-03-31T15:50:44.049 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../068-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:44.093 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:44.157 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-openssl. 2024-03-31T15:50:44.167 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-31T15:50:44.232 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-31T15:50:44.412 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-fuse. 2024-03-31T15:50:44.427 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:44.470 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:44.569 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr. 2024-03-31T15:50:44.579 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:44.619 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:44.874 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mds. 2024-03-31T15:50:44.888 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:44.932 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:44.999 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mon. 2024-03-31T15:50:45.012 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:45.048 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:45.318 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package cephadm. 2024-03-31T15:50:45.331 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../071-cephadm_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:45.376 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:45.629 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-31T15:50:45.642 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-31T15:50:45.678 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-osd. 2024-03-31T15:50:45.687 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-31T15:50:45.688 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:45.736 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:46.015 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-31T15:50:46.027 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:50:46.056 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:50:46.591 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph. 2024-03-31T15:50:46.602 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../068-ceph_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:46.641 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:46.960 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-fuse. 2024-03-31T15:50:46.971 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:47.010 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:47.086 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-babel. 2024-03-31T15:50:47.100 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:50:47.144 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:50:47.374 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mds. 2024-03-31T15:50:47.385 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:47.423 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:47.481 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jinja2. 2024-03-31T15:50:47.493 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-31T15:50:47.530 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:50:47.827 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephadm. 2024-03-31T15:50:47.838 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../071-cephadm_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:47.876 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:47.883 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-31T15:50:47.896 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:47.933 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:48.197 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-31T15:50:48.208 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-31T15:50:48.255 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-31T15:50:48.262 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-31T15:50:48.274 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-31T15:50:48.336 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-31T15:50:48.601 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-31T15:50:48.613 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:50:48.650 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:50:48.673 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-routes. 2024-03-31T15:50:48.686 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-31T15:50:48.723 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:50:49.076 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-31T15:50:49.092 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:49.134 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:49.690 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-babel. 2024-03-31T15:50:49.700 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-31T15:50:49.748 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:50:50.111 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jinja2. 2024-03-31T15:50:50.122 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-31T15:50:50.161 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:50:50.431 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-31T15:50:50.444 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-31T15:50:50.472 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:50:50.498 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-31T15:50:50.508 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:50.548 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:50.895 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-31T15:50:50.905 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-31T15:50:50.944 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-31T15:50:50.993 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-joblib. 2024-03-31T15:50:51.006 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-31T15:50:51.051 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:50:51.248 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-routes. 2024-03-31T15:50:51.258 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-31T15:50:51.356 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:50:51.413 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-31T15:50:51.426 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-31T15:50:51.581 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:50:51.686 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-31T15:50:51.697 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:51.744 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:51.834 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-sklearn. 2024-03-31T15:50:51.848 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-31T15:50:51.892 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:50:52.526 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-31T15:50:52.546 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:52.584 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:53.242 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-31T15:50:53.251 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-31T15:50:53.292 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:50:53.572 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cachetools. 2024-03-31T15:50:53.586 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-31T15:50:53.621 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-31T15:50:53.764 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-joblib. 2024-03-31T15:50:53.772 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-31T15:50:53.814 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:50:53.937 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-31T15:50:53.951 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-31T15:50:53.999 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:50:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-31T15:50:54.170 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-31T15:50:54.319 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-31T15:50:54.330 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:50:54.332 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-31T15:50:54.377 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:50:54.633 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn. 2024-03-31T15:50:54.643 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-31T15:50:54.692 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:50:54.714 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rsa. 2024-03-31T15:50:54.728 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-31T15:50:54.772 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-31T15:50:55.026 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-google-auth. 2024-03-31T15:50:55.040 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-31T15:50:55.083 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-31T15:50:55.320 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-31T15:50:55.330 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:55.370 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:55.429 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-31T15:50:55.439 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-31T15:50:55.478 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:50:55.798 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-websocket. 2024-03-31T15:50:55.808 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-31T15:50:55.873 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-31T15:50:56.350 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cachetools. 2024-03-31T15:50:56.360 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-31T15:50:56.402 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-31T15:50:56.408 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-31T15:50:56.411 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-31T15:50:56.476 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:50:56.686 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-31T15:50:56.697 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-31T15:50:56.736 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:50:57.065 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-31T15:50:57.076 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-31T15:50:57.116 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:50:57.157 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-31T15:50:57.166 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:57.462 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rsa. 2024-03-31T15:50:57.472 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-31T15:50:57.511 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-31T15:50:57.518 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-31T15:50:57.527 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-31T15:50:57.568 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:50:57.816 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-google-auth. 2024-03-31T15:50:57.827 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-31T15:50:57.874 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-31T15:50:58.054 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-31T15:50:58.063 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:50:58.104 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:50:58.329 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-31T15:50:58.340 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-31T15:50:58.348 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package jq. 2024-03-31T15:50:58.360 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:50:58.379 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:50:58.390 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-31T15:50:58.601 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package socat. 2024-03-31T15:50:58.614 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-31T15:50:58.651 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:50:58.683 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-websocket. 2024-03-31T15:50:58.694 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-31T15:50:58.741 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-31T15:50:58.946 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package xmlstarlet. 2024-03-31T15:50:58.957 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-31T15:50:58.995 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:50:59.063 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-31T15:50:59.073 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-31T15:50:59.112 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:50:59.308 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-test. 2024-03-31T15:50:59.318 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:50:59.358 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:50:59.769 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-31T15:50:59.779 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:50:59.827 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:00.131 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-31T15:51:00.141 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-31T15:51:00.181 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:51:00.552 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-31T15:51:00.561 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:51:00.601 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:51:00.856 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package jq. 2024-03-31T15:51:00.866 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-31T15:51:00.906 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-31T15:51:01.151 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package socat. 2024-03-31T15:51:01.161 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-31T15:51:01.200 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:51:01.396 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-volume. 2024-03-31T15:51:01.407 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:01.445 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:01.471 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package xmlstarlet. 2024-03-31T15:51:01.482 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-31T15:51:01.521 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:51:01.841 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-31T15:51:01.853 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:01.891 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:01.902 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-test. 2024-03-31T15:51:01.912 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:01.952 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:02.202 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-colorama. 2024-03-31T15:51:02.214 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-31T15:51:02.277 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-31T15:51:02.538 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package cephfs-shell. 2024-03-31T15:51:02.549 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:02.587 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:02.807 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package cephfs-top. 2024-03-31T15:51:02.819 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:02.883 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:03.126 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-31T15:51:03.138 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:03.176 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:03.470 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-31T15:51:03.485 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-31T15:51:03.545 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:51:03.883 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-31T15:51:03.896 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-31T15:51:03.933 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:51:04.270 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package nvme-cli. 2024-03-31T15:51:04.282 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-31T15:51:04.311 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:51:04.416 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-volume. 2024-03-31T15:51:04.427 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:04.466 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:04.711 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package pkg-config. 2024-03-31T15:51:04.723 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-31T15:51:04.794 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:51:04.804 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-31T15:51:04.815 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:04.871 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:05.047 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-31T15:51:05.059 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-31T15:51:05.096 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:51:05.184 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-colorama. 2024-03-31T15:51:05.195 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-31T15:51:05.234 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-31T15:51:05.495 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephfs-shell. 2024-03-31T15:51:05.506 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:05.553 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:05.558 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-31T15:51:05.571 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-31T15:51:05.616 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-31T15:51:05.798 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephfs-top. 2024-03-31T15:51:05.809 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2574-gd2271579-1jammy_all.deb ... 2024-03-31T15:51:05.848 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:05.877 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pastescript. 2024-03-31T15:51:05.890 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-31T15:51:05.927 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-31T15:51:06.102 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-31T15:51:06.113 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:06.152 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:06.239 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pluggy. 2024-03-31T15:51:06.252 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-31T15:51:06.280 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:51:06.522 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-31T15:51:06.531 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-31T15:51:06.571 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:51:06.625 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-psutil. 2024-03-31T15:51:06.638 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-31T15:51:06.675 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-31T15:51:06.893 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-31T15:51:06.903 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-31T15:51:06.944 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:51:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-py. 2024-03-31T15:51:06.993 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-31T15:51:07.036 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-31T15:51:07.299 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package nvme-cli. 2024-03-31T15:51:07.309 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-31T15:51:07.348 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:51:07.389 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pygments. 2024-03-31T15:51:07.403 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-31T15:51:07.448 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:51:07.789 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package pkg-config. 2024-03-31T15:51:07.800 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-31T15:51:07.838 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:51:07.952 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-31T15:51:07.966 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-31T15:51:08.010 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:51:08.159 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-31T15:51:08.169 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-31T15:51:08.217 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:51:08.321 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-toml. 2024-03-31T15:51:08.334 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-31T15:51:08.379 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-31T15:51:08.657 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pytest. 2024-03-31T15:51:08.670 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-31T15:51:08.704 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-31T15:51:08.706 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:51:08.714 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-31T15:51:08.753 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-31T15:51:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastescript. 2024-03-31T15:51:09.044 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-simplejson. 2024-03-31T15:51:09.055 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-31T15:51:09.057 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-31T15:51:09.110 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:51:09.128 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-31T15:51:09.380 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-31T15:51:09.393 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-31T15:51:09.428 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pluggy. 2024-03-31T15:51:09.438 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:51:09.440 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-31T15:51:09.478 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:51:09.758 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-psutil. 2024-03-31T15:51:09.769 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-31T15:51:09.807 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-31T15:51:10.119 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-py. 2024-03-31T15:51:10.125 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package radosgw. 2024-03-31T15:51:10.130 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-31T15:51:10.138 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../122-radosgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:10.167 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:10.168 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-31T15:51:10.456 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pygments. 2024-03-31T15:51:10.468 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-31T15:51:10.506 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:51:10.778 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package rbd-fuse. 2024-03-31T15:51:10.792 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:10.836 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:10.961 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-31T15:51:10.972 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-31T15:51:11.019 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:51:11.146 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-31T15:51:11.297 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-toml. 2024-03-31T15:51:11.308 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-31T15:51:11.347 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-31T15:51:11.400 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:51:11.643 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pytest. 2024-03-31T15:51:11.655 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-31T15:51:11.693 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:51:11.753 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:51:12.012 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-31T15:51:12.030 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplejson. 2024-03-31T15:51:12.041 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-31T15:51:12.080 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:51:12.333 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-31T15:51:12.345 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-31T15:51:12.383 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:51:12.403 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-31T15:51:13.016 INFO:teuthology.orchestra.run.smithi102.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-31T15:51:13.042 INFO:teuthology.orchestra.run.smithi102.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-31T15:51:13.063 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package radosgw. 2024-03-31T15:51:13.074 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../122-radosgw_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:13.087 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:13.113 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:13.817 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package rbd-fuse. 2024-03-31T15:51:13.829 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2574-gd2271579-1jammy_amd64.deb ... 2024-03-31T15:51:13.867 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:13.890 INFO:teuthology.orchestra.run.smithi102.stdout:Adding system user cephadm....done 2024-03-31T15:51:14.027 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:51:14.170 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-31T15:51:14.312 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:51:14.449 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-31T15:51:14.550 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-31T15:51:14.571 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:51:14.697 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:51:14.802 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-31T15:51:14.957 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-31T15:51:15.207 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:51:15.208 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-31T15:51:15.309 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-31T15:51:15.611 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:51:15.766 INFO:teuthology.orchestra.run.smithi090.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-31T15:51:15.824 INFO:teuthology.orchestra.run.smithi090.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-31T15:51:15.894 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:15.979 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-31T15:51:16.272 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-31T15:51:16.439 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user cephadm....done 2024-03-31T15:51:16.540 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-31T15:51:16.584 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-31T15:51:16.808 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:51:16.861 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-31T15:51:17.051 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:51:17.122 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-31T15:51:17.248 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-31T15:51:17.444 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:51:17.499 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-31T15:51:17.712 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:51:17.742 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-31T15:51:17.902 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-31T15:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:18.212 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-31T15:51:18.247 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:51:18.373 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:51:18.482 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-31T15:51:18.582 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-31T15:51:18.759 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:51:18.833 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-31T15:51:18.868 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:51:19.093 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-31T15:51:19.161 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:51:19.567 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-31T15:51:19.599 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-31T15:51:19.661 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-31T15:51:19.664 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:51:19.815 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-31T15:51:19.929 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-31T15:51:20.191 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:51:20.197 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-31T15:51:20.292 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-31T15:51:20.458 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-31T15:51:20.577 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:51:20.701 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-argparse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:20.845 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:51:20.969 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-31T15:51:21.094 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-31T15:51:21.162 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up zip (3.0-12build2) ... 2024-03-31T15:51:21.220 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-31T15:51:21.288 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:51:21.480 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-31T15:51:21.606 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-31T15:51:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-31T15:51:21.974 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:51:22.258 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:51:22.349 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-31T15:51:22.359 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:51:22.382 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-31T15:51:22.446 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-31T15:51:22.485 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:51:22.572 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-31T15:51:22.836 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-31T15:51:22.948 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-31T15:51:23.074 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-31T15:51:23.238 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-31T15:51:23.344 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-31T15:51:23.497 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:51:23.603 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-31T15:51:23.891 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:51:23.921 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up zip (3.0-12build2) ... 2024-03-31T15:51:24.047 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-31T15:51:24.184 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-31T15:51:24.501 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:51:24.733 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-31T15:51:24.853 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:51:25.009 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-31T15:51:25.120 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-31T15:51:25.121 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:51:25.246 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-31T15:51:25.264 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:25.556 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-31T15:51:25.572 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-idna (3.3-1) ... 2024-03-31T15:51:25.907 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:51:25.957 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-31T15:51:26.217 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-31T15:51:26.635 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-31T15:51:26.921 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-31T15:51:27.172 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:51:27.239 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-31T15:51:27.423 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:27.549 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-31T15:51:27.590 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-31T15:51:27.826 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:51:27.858 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-31T15:51:27.951 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:51:27.984 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:28.077 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:51:28.269 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-idna (3.3-1) ... 2024-03-31T15:51:28.413 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:51:28.605 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-31T15:51:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-31T15:51:28.939 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:29.098 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:51:29.400 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:51:29.651 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-31T15:51:29.870 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-31T15:51:29.919 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-31T15:51:30.123 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:30.187 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:51:30.258 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-31T15:51:30.471 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-31T15:51:30.567 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-31T15:51:30.677 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-31T15:51:30.747 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:51:30.778 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-31T15:51:30.898 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-31T15:51:31.079 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-31T15:51:31.174 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:51:31.284 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:51:31.347 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-31T15:51:31.568 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:51:31.608 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:31.734 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-31T15:51:31.878 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:51:32.044 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-31T15:51:32.154 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:51:32.295 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-31T15:51:32.563 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-31T15:51:32.571 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:51:32.731 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:51:32.811 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-31T15:51:32.823 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-31T15:51:33.058 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:51:33.083 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-31T15:51:33.159 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:51:33.360 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-31T15:51:33.502 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-31T15:51:33.535 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:51:33.770 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-31T15:51:33.803 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-31T15:51:33.896 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-31T15:51:33.954 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-31T15:51:34.190 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-31T15:51:34.230 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:34.348 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:51:34.499 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-31T15:51:34.658 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:51:34.768 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-31T15:51:34.936 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-31T15:51:34.959 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:51:35.037 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-31T15:51:35.125 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-31T15:51:35.235 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-31T15:51:35.372 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-31T15:51:35.446 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-31T15:51:35.498 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-31T15:51:35.723 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-31T15:51:35.892 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-31T15:51:36.015 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:51:36.161 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-31T15:51:36.275 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:51:36.295 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-31T15:51:36.555 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-31T15:51:36.652 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-31T15:51:36.681 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-31T15:51:36.945 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:51:37.008 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-31T15:51:37.317 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-31T15:51:37.322 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:51:37.424 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:37.534 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:37.601 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-31T15:51:37.693 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:51:37.848 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-31T15:51:38.021 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:38.115 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-31T15:51:38.146 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:38.298 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-31T15:51:38.392 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-31T15:51:38.591 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:38.660 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-31T15:51:38.708 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:39.022 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-31T15:51:39.060 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:51:39.061 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:51:39.340 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-31T15:51:39.661 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:39.684 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-31T15:51:39.787 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:39.810 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librados2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:39.913 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:39.920 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:40.039 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:40.039 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-31T15:51:40.190 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:51:40.376 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:40.527 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libradosstriper1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:40.645 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-31T15:51:40.913 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librbd1 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.040 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-modules-core (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.166 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.377 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:51:41.377 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-31T15:51:41.469 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.595 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.721 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:41.847 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.006 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.010 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs-dev (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.119 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rados (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.132 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.220 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librgw2 (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.346 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephfs-top (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:42.472 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-31T15:51:42.643 INFO:teuthology.orchestra.run.smithi102.stdout:Adding group ceph....done 2024-03-31T15:51:42.978 INFO:teuthology.orchestra.run.smithi102.stdout:Adding system user ceph....done 2024-03-31T15:51:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:Setting system user ceph properties....done 2024-03-31T15:51:43.150 INFO:teuthology.orchestra.run.smithi102.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-31T15:51:43.291 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-31T15:51:43.678 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-31T15:51:43.745 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rbd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:43.845 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up rbd-fuse (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:43.963 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:44.090 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cephfs (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:44.215 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephfs-shell (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:44.343 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-common (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:44.375 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:44.593 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:51:44.593 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:51:44.888 INFO:teuthology.orchestra.run.smithi090.stdout:Adding group ceph....done 2024-03-31T15:51:45.194 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:45.257 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user ceph....done 2024-03-31T15:51:45.320 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:45.393 INFO:teuthology.orchestra.run.smithi090.stdout:Setting system user ceph properties....done 2024-03-31T15:51:45.412 INFO:teuthology.orchestra.run.smithi090.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-31T15:51:45.554 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-31T15:51:45.825 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:51:45.825 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:51:45.934 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-31T15:51:46.438 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:46.605 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephfs-mirror (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:46.791 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-31T15:51:46.813 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:51:46.813 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-31T15:51:47.439 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-test (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:47.507 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:47.565 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up radosgw (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:47.762 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:51:47.762 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:51:48.125 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:51:48.125 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-31T15:51:48.409 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:48.622 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:51:48.622 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:51:48.734 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-base (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:49.091 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-31T15:51:49.286 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:49.748 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:51:49.748 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:51:49.754 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mds (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:49.984 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:51:49.984 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-31T15:51:50.354 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:50.471 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:50.690 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:50.847 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:51:50.848 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:51:50.909 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:51:50.909 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-31T15:51:51.465 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:51.609 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:51.610 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-osd (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:51.709 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:51.844 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:52.160 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:51:52.161 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-31T15:51:52.222 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:51:52.330 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:51:52.466 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:51:52.873 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:53.000 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:53.151 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mon (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:53.362 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:51:53.362 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-31T15:51:53.584 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:51:54.019 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-cephadm (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:54.170 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:54.279 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-dashboard (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:51:54.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-31T15:51:54.393 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-03-31T15:51:54.407 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-volume (19.0.0-2574-gd2271579-1jammy) ... 2024-03-31T15:51:54.565 INFO:teuthology.orchestra.run.smithi102.stdout:Building dependency tree... 2024-03-31T15:51:54.566 INFO:teuthology.orchestra.run.smithi102.stdout:Reading state information... 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:51:54.731 INFO:teuthology.orchestra.run.smithi102.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:51:54.732 INFO:teuthology.orchestra.run.smithi102.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:51:54.732 INFO:teuthology.orchestra.run.smithi102.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:51:54.732 INFO:teuthology.orchestra.run.smithi102.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:51:54.732 INFO:teuthology.orchestra.run.smithi102.stdout:The following additional packages will be installed: 2024-03-31T15:51:54.733 INFO:teuthology.orchestra.run.smithi102.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-31T15:51:54.733 INFO:teuthology.orchestra.run.smithi102.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-31T15:51:54.733 INFO:teuthology.orchestra.run.smithi102.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-31T15:51:54.733 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-31T15:51:54.733 INFO:teuthology.orchestra.run.smithi102.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:51:54.734 INFO:teuthology.orchestra.run.smithi102.stdout:Suggested packages: 2024-03-31T15:51:54.734 INFO:teuthology.orchestra.run.smithi102.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-31T15:51:54.734 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-31T15:51:54.734 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-31T15:51:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:51:54.788 INFO:teuthology.orchestra.run.smithi102.stdout:The following NEW packages will be installed: 2024-03-31T15:51:54.788 INFO:teuthology.orchestra.run.smithi102.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-31T15:51:54.788 INFO:teuthology.orchestra.run.smithi102.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-31T15:51:54.789 INFO:teuthology.orchestra.run.smithi102.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-31T15:51:54.789 INFO:teuthology.orchestra.run.smithi102.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-31T15:51:54.789 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-31T15:51:54.789 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-31T15:51:54.789 INFO:teuthology.orchestra.run.smithi102.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:51:54.790 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages will be upgraded: 2024-03-31T15:51:54.791 INFO:teuthology.orchestra.run.smithi102.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-31T15:51:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:4 upgraded, 39 newly installed, 0 to remove and 297 not upgraded. 2024-03-31T15:51:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:Need to get 55.7 MB of archives. 2024-03-31T15:51:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-31T15:51:54.899 INFO:teuthology.orchestra.run.smithi102.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-31T15:51:54.910 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:51:55.063 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:51:55.118 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-31T15:51:55.126 INFO:teuthology.orchestra.run.smithi102.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-31T15:51:55.183 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-31T15:51:55.183 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-31T15:51:55.184 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-31T15:51:55.203 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-31T15:51:55.204 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-31T15:51:55.207 INFO:teuthology.orchestra.run.smithi102.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-31T15:51:55.209 INFO:teuthology.orchestra.run.smithi102.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-31T15:51:55.209 INFO:teuthology.orchestra.run.smithi102.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-31T15:51:55.210 INFO:teuthology.orchestra.run.smithi102.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-31T15:51:55.236 INFO:teuthology.orchestra.run.smithi102.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-31T15:51:55.236 INFO:teuthology.orchestra.run.smithi102.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-31T15:51:55.236 INFO:teuthology.orchestra.run.smithi102.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-31T15:51:55.263 INFO:teuthology.orchestra.run.smithi102.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-31T15:51:55.782 INFO:teuthology.orchestra.run.smithi102.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-31T15:51:55.832 INFO:teuthology.orchestra.run.smithi102.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-31T15:51:55.832 INFO:teuthology.orchestra.run.smithi102.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-31T15:51:55.835 INFO:teuthology.orchestra.run.smithi102.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-31T15:51:55.836 INFO:teuthology.orchestra.run.smithi102.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-31T15:51:55.840 INFO:teuthology.orchestra.run.smithi102.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-31T15:51:55.842 INFO:teuthology.orchestra.run.smithi102.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-31T15:51:55.860 INFO:teuthology.orchestra.run.smithi102.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-31T15:51:55.861 INFO:teuthology.orchestra.run.smithi102.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-31T15:51:55.908 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:51:55.911 INFO:teuthology.orchestra.run.smithi102.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-31T15:51:55.911 INFO:teuthology.orchestra.run.smithi102.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-31T15:51:55.940 INFO:teuthology.orchestra.run.smithi102.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-31T15:51:56.052 INFO:teuthology.orchestra.run.smithi102.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-31T15:51:56.056 INFO:teuthology.orchestra.run.smithi102.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-31T15:51:56.056 INFO:teuthology.orchestra.run.smithi102.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-31T15:51:56.057 INFO:teuthology.orchestra.run.smithi102.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-31T15:51:56.057 INFO:teuthology.orchestra.run.smithi102.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-31T15:51:56.068 INFO:teuthology.orchestra.run.smithi102.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-31T15:51:56.070 INFO:teuthology.orchestra.run.smithi102.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-31T15:51:56.496 INFO:teuthology.orchestra.run.smithi102.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-31T15:51:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-31T15:51:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-31T15:51:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-31T15:51:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-31T15:51:56.497 INFO:teuthology.orchestra.run.smithi102.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-31T15:51:56.512 INFO:teuthology.orchestra.run.smithi102.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-31T15:51:56.512 INFO:teuthology.orchestra.run.smithi102.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-31T15:51:56.700 INFO:teuthology.orchestra.run.smithi090.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:51:56.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-31T15:51:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2024-03-31T15:51:56.883 INFO:teuthology.orchestra.run.smithi102.stdout:Preconfiguring packages ... 2024-03-31T15:51:56.950 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2024-03-31T15:51:56.951 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2024-03-31T15:51:57.032 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 55.7 MB in 2s (32.5 MB/s) 2024-03-31T15:51:57.122 INFO:teuthology.orchestra.run.smithi090.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-31T15:51:57.122 INFO:teuthology.orchestra.run.smithi090.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-31T15:51:57.122 INFO:teuthology.orchestra.run.smithi090.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-31T15:51:57.122 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2024-03-31T15:51:57.122 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-31T15:51:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-31T15:51:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-31T15:51:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-31T15:51:57.123 INFO:teuthology.orchestra.run.smithi090.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-31T15:51:57.123 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-31T15:51:57.125 INFO:teuthology.orchestra.run.smithi090.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:51:57.126 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2024-03-31T15:51:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-31T15:51:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-03-31T15:51:57.126 INFO:teuthology.orchestra.run.smithi090.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-03-31T15:51:57.178 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2024-03-31T15:51:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-31T15:51:57.179 INFO:teuthology.orchestra.run.smithi090.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-31T15:51:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-31T15:51:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-31T15:51:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-31T15:51:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-31T15:51:57.180 INFO:teuthology.orchestra.run.smithi090.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-31T15:51:57.181 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2024-03-31T15:51:57.182 INFO:teuthology.orchestra.run.smithi090.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-03-31T15:51:57.245 INFO:teuthology.orchestra.run.smithi090.stdout:4 upgraded, 39 newly installed, 0 to remove and 297 not upgraded. 2024-03-31T15:51:57.245 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 55.7 MB of archives. 2024-03-31T15:51:57.245 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-31T15:51:57.246 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.3 [1,063 kB] 2024-03-31T15:51:57.320 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:51:57.323 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:51:57.472 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-31T15:51:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-31T15:51:57.498 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:51:57.554 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.3 [33.1 kB] 2024-03-31T15:51:57.555 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.3 [24.4 kB] 2024-03-31T15:51:57.555 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-31T15:51:57.571 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-31T15:51:57.572 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-31T15:51:57.574 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-31T15:51:57.578 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-31T15:51:57.578 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-31T15:51:57.578 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-31T15:51:57.580 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-31T15:51:57.599 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-31T15:51:57.600 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-31T15:51:57.601 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-31T15:51:57.911 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-31T15:51:57.913 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-31T15:51:57.914 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-31T15:51:57.917 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-31T15:51:57.918 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-31T15:51:57.923 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-31T15:51:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-31T15:51:57.933 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-31T15:51:57.933 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-31T15:51:57.957 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-31T15:51:57.963 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-31T15:51:57.974 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-31T15:51:58.001 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-31T15:51:58.005 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-31T15:51:58.006 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-31T15:51:58.007 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-31T15:51:58.007 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-31T15:51:58.023 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-31T15:51:58.028 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-31T15:51:58.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-31T15:51:58.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-31T15:51:58.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-31T15:51:58.245 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-31T15:51:58.246 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-31T15:51:58.246 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-31T15:51:58.246 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-31T15:51:58.260 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-31T15:51:58.261 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-31T15:51:58.637 INFO:teuthology.orchestra.run.smithi090.stdout:Preconfiguring packages ... 2024-03-31T15:51:58.791 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 55.7 MB in 1s (52.0 MB/s) 2024-03-31T15:51:59.081 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:51:59.085 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:51:59.250 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking util-linux (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:51:59.275 INFO:teuthology.orchestra.run.smithi102.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-31T15:51:59.565 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package flex. 2024-03-31T15:51:59.636 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:51:59.639 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-31T15:51:59.680 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-31T15:51:59.871 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up util-linux (2.37.2-4ubuntu3.3) ... 2024-03-31T15:52:00.316 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:52:00.495 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-31T15:52:00.884 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:00.920 INFO:teuthology.orchestra.run.smithi090.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-03-31T15:52:01.155 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package flex. 2024-03-31T15:52:01.226 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140915 files and directories currently installed.) 2024-03-31T15:52:01.229 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-31T15:52:01.270 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-31T15:52:01.446 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:52:01.449 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:52:01.578 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:52:01.733 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:52:01.924 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-31T15:52:01.965 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:52:02.101 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:52:02.416 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:52:02.435 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:02.818 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package bison. 2024-03-31T15:52:02.893 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:52:02.897 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-31T15:52:02.945 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:52:02.973 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:52:02.977 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:52:03.104 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:52:03.519 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.3_amd64.deb ... 2024-03-31T15:52:03.654 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.3) over (2.37.2-4ubuntu3) ... 2024-03-31T15:52:03.831 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-31T15:52:03.846 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-31T15:52:03.889 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:52:03.979 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:52:04.209 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package dump. 2024-03-31T15:52:04.224 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-31T15:52:04.259 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking dump (0.4b47-3) ... 2024-03-31T15:52:04.332 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package bison. 2024-03-31T15:52:04.407 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140988 files and directories currently installed.) 2024-03-31T15:52:04.411 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-31T15:52:04.459 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:52:04.563 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package indent. 2024-03-31T15:52:04.578 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:04.621 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:52:04.882 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-31T15:52:04.897 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-31T15:52:04.920 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-31T15:52:04.935 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-31T15:52:04.940 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:52:04.970 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:52:05.244 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-31T15:52:05.259 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-31T15:52:05.294 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:52:05.298 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package dump. 2024-03-31T15:52:05.314 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-31T15:52:05.356 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking dump (0.4b47-3) ... 2024-03-31T15:52:05.556 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-31T15:52:05.571 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-31T15:52:05.620 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package indent. 2024-03-31T15:52:05.630 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:52:05.636 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:05.670 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:52:05.961 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-31T15:52:05.965 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-31T15:52:05.974 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-31T15:52:05.981 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-31T15:52:06.017 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:06.023 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:52:06.278 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-31T15:52:06.294 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-31T15:52:06.328 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:06.378 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-31T15:52:06.394 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-31T15:52:06.436 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:52:06.589 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libjson-perl. 2024-03-31T15:52:06.605 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-31T15:52:06.647 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-31T15:52:06.690 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-31T15:52:06.706 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-31T15:52:06.748 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:52:06.984 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-31T15:52:06.999 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-31T15:52:07.042 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:52:07.086 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-31T15:52:07.102 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-31T15:52:07.144 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:07.390 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-31T15:52:07.406 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-31T15:52:07.440 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:07.718 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjson-perl. 2024-03-31T15:52:07.733 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-31T15:52:07.776 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-31T15:52:08.105 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-31T15:52:08.120 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-31T15:52:08.163 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:52:08.421 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-31T15:52:08.436 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:52:08.479 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:08.808 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-31T15:52:08.823 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:52:08.866 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:09.185 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-31T15:52:09.201 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:52:09.243 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:09.605 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-31T15:52:09.620 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-31T15:52:09.655 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:52:09.851 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-31T15:52:09.867 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:52:09.909 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:10.000 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-31T15:52:10.015 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:52:10.058 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:52:10.171 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-31T15:52:10.186 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-31T15:52:10.229 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:10.394 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-31T15:52:10.410 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:10.452 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:10.557 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-31T15:52:10.572 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:52:10.616 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:10.806 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-31T15:52:10.821 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:10.864 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:10.978 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-31T15:52:10.994 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-31T15:52:11.070 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:52:11.175 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-31T15:52:11.190 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:11.225 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:11.407 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-31T15:52:11.423 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:52:11.444 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-31T15:52:11.459 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:52:11.465 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:52:11.502 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:11.812 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-31T15:52:11.828 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:11.878 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:12.031 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-31T15:52:12.047 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-31T15:52:12.089 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:52:12.282 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-31T15:52:12.298 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:12.373 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:12.443 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package modemmanager. 2024-03-31T15:52:12.458 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:52:12.492 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:12.703 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-31T15:52:12.719 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:12.786 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:13.097 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-31T15:52:13.113 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-31T15:52:13.164 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:13.233 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package network-manager. 2024-03-31T15:52:13.248 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:52:13.299 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:52:13.801 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-31T15:52:13.816 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-31T15:52:13.859 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:52:14.083 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ppp. 2024-03-31T15:52:14.098 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-31T15:52:14.289 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package modemmanager. 2024-03-31T15:52:14.304 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-31T15:52:14.358 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:14.440 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:52:14.738 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package pptp-linux. 2024-03-31T15:52:14.752 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-31T15:52:14.796 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-31T15:52:15.031 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package network-manager. 2024-03-31T15:52:15.046 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-31T15:52:15.082 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-31T15:52:15.096 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-31T15:52:15.097 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:52:15.143 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-31T15:52:15.385 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-31T15:52:15.399 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-31T15:52:15.460 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-client-common (238) ... 2024-03-31T15:52:15.749 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ppp. 2024-03-31T15:52:15.764 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-31T15:52:15.764 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-31T15:52:15.778 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:15.822 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:16.106 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:52:16.369 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-common. 2024-03-31T15:52:16.383 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-31T15:52:16.436 INFO:teuthology.orchestra.run.smithi102.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-31T15:52:16.459 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-common (238) ... 2024-03-31T15:52:16.573 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package pptp-linux. 2024-03-31T15:52:16.588 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-31T15:52:16.632 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-31T15:52:16.910 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-14. 2024-03-31T15:52:16.924 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:16.943 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-31T15:52:16.957 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-31T15:52:16.970 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:17.002 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-31T15:52:17.280 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-31T15:52:17.294 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-31T15:52:17.330 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-client-common (238) ... 2024-03-31T15:52:17.627 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-31T15:52:17.642 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:17.685 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:18.062 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql. 2024-03-31T15:52:18.077 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-31T15:52:18.121 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql (14+238) ... 2024-03-31T15:52:18.323 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-client. 2024-03-31T15:52:18.338 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-31T15:52:18.366 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-common. 2024-03-31T15:52:18.373 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-client (14+238) ... 2024-03-31T15:52:18.380 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-31T15:52:18.433 INFO:teuthology.orchestra.run.smithi090.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-31T15:52:18.466 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-common (238) ... 2024-03-31T15:52:18.675 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-31T15:52:18.690 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-31T15:52:18.733 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-31T15:52:18.933 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-14. 2024-03-31T15:52:18.948 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-31T15:52:18.994 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:19.069 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package tcl8.6. 2024-03-31T15:52:19.085 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-31T15:52:19.128 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:52:19.431 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package tcl. 2024-03-31T15:52:19.446 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-31T15:52:19.485 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-31T15:52:19.767 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package wpasupplicant. 2024-03-31T15:52:19.782 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-31T15:52:19.858 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:52:20.123 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql. 2024-03-31T15:52:20.138 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-31T15:52:20.171 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-31T15:52:20.181 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql (14+238) ... 2024-03-31T15:52:20.186 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-31T15:52:20.229 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-31T15:52:20.426 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-client. 2024-03-31T15:52:20.440 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-31T15:52:20.484 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-client (14+238) ... 2024-03-31T15:52:20.649 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-31T15:52:20.665 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-31T15:52:20.716 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:52:20.728 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-31T15:52:20.742 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-31T15:52:20.786 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-31T15:52:21.009 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-client-common (238) ... 2024-03-31T15:52:21.123 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package tcl8.6. 2024-03-31T15:52:21.138 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-31T15:52:21.173 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:52:21.239 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-31T15:52:21.365 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:52:21.491 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:21.617 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:52:21.768 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:52:21.894 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:21.993 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package tcl. 2024-03-31T15:52:22.008 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-31T15:52:22.056 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-31T15:52:22.398 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package wpasupplicant. 2024-03-31T15:52:22.413 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-31T15:52:22.448 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:52:22.811 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-31T15:52:22.826 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-31T15:52:22.827 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-31T15:52:22.869 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-31T15:52:22.916 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:52:23.240 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-31T15:52:23.255 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-31T15:52:23.306 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:52:23.310 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:52:23.310 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:52:23.600 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-client-common (238) ... 2024-03-31T15:52:23.789 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-31T15:52:23.956 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-31T15:52:23.978 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:52:24.084 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:24.103 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:52:24.210 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-31T15:52:24.246 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:24.336 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-31T15:52:24.381 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-31T15:52:24.462 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:24.498 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:52:24.608 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:24.709 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:52:24.860 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:52:25.401 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-31T15:52:25.503 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-31T15:52:25.607 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:52:25.708 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up dump (0.4b47-3) ... 2024-03-31T15:52:25.802 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-31T15:52:25.856 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:52:25.856 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-31T15:52:25.876 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:26.002 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:52:26.145 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-31T15:52:26.271 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-31T15:52:26.405 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:52:26.495 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-31T15:52:26.548 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.3) ... 2024-03-31T15:52:26.557 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:52:26.674 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-31T15:52:26.699 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:52:26.834 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-31T15:52:26.842 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:52:26.968 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:27.094 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-31T15:52:27.288 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-client (14+238) ... 2024-03-31T15:52:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:52:27.445 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-31T15:52:27.554 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-31T15:52:27.739 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:27.865 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-31T15:52:27.991 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-31T15:52:28.028 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:28.028 INFO:teuthology.orchestra.run.smithi102.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-31T15:52:28.028 INFO:teuthology.orchestra.run.smithi102.stdout:which means they are currently configured by ifupdown: 2024-03-31T15:52:28.029 INFO:teuthology.orchestra.run.smithi102.stdout:- enp3s0f1 2024-03-31T15:52:28.029 INFO:teuthology.orchestra.run.smithi102.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-31T15:52:28.029 INFO:teuthology.orchestra.run.smithi102.stdout:remove their configuration from /etc/network/interfaces. 2024-03-31T15:52:28.029 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:28.161 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-31T15:52:28.543 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-31T15:52:28.682 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-31T15:52:28.791 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up dump (0.4b47-3) ... 2024-03-31T15:52:28.886 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-31T15:52:28.947 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-31T15:52:28.951 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:29.077 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-31T15:52:29.204 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-31T15:52:29.331 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-31T15:52:29.440 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-31T15:52:29.513 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-31T15:52:29.583 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-31T15:52:29.741 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:29.743 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-31T15:52:29.876 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:29.919 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-31T15:52:30.002 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-31T15:52:30.045 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:30.128 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-common (238) ... 2024-03-31T15:52:30.173 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-31T15:52:30.316 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-client (14+238) ... 2024-03-31T15:52:30.425 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-31T15:52:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:31.000 INFO:teuthology.orchestra.run.smithi090.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-31T15:52:31.000 INFO:teuthology.orchestra.run.smithi090.stdout:which means they are currently configured by ifupdown: 2024-03-31T15:52:31.000 INFO:teuthology.orchestra.run.smithi090.stdout:- enp3s0f1 2024-03-31T15:52:31.001 INFO:teuthology.orchestra.run.smithi090.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-31T15:52:31.001 INFO:teuthology.orchestra.run.smithi090.stdout:remove their configuration from /etc/network/interfaces. 2024-03-31T15:52:31.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:31.080 INFO:teuthology.orchestra.run.smithi102.stdout:Adding user postgres to group ssl-cert 2024-03-31T15:52:31.133 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-31T15:52:31.332 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:31.332 INFO:teuthology.orchestra.run.smithi102.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-31T15:52:31.550 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-31T15:52:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-31T15:52:31.559 INFO:teuthology.orchestra.run.smithi102.stdout:Removing obsolete dictionary files: 2024-03-31T15:52:32.003 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-31T15:52:32.264 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-31T15:52:32.712 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-31T15:52:32.721 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:32.855 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-31T15:52:32.982 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-31T15:52:33.083 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-common (238) ... 2024-03-31T15:52:33.120 INFO:teuthology.orchestra.run.smithi102.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-31T15:52:33.138 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:This user must also own the server process. 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:The database cluster will be initialized with locale "en_US". 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:The default text search configuration will be set to "english". 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout:Data page checksums are disabled. 2024-03-31T15:52:33.188 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:33.189 INFO:teuthology.orchestra.run.smithi102.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-31T15:52:33.189 INFO:teuthology.orchestra.run.smithi102.stdout:creating subdirectories ... ok 2024-03-31T15:52:33.189 INFO:teuthology.orchestra.run.smithi102.stdout:selecting dynamic shared memory implementation ... posix 2024-03-31T15:52:33.212 INFO:teuthology.orchestra.run.smithi102.stdout:selecting default max_connections ... 100 2024-03-31T15:52:33.246 INFO:teuthology.orchestra.run.smithi102.stdout:selecting default shared_buffers ... 128MB 2024-03-31T15:52:33.260 INFO:teuthology.orchestra.run.smithi102.stdout:selecting default time zone ... Etc/UTC 2024-03-31T15:52:33.261 INFO:teuthology.orchestra.run.smithi102.stdout:creating configuration files ... ok 2024-03-31T15:52:33.423 INFO:teuthology.orchestra.run.smithi102.stdout:running bootstrap script ... ok 2024-03-31T15:52:34.025 INFO:teuthology.orchestra.run.smithi102.stdout:performing post-bootstrap initialization ... ok 2024-03-31T15:52:34.038 INFO:teuthology.orchestra.run.smithi090.stdout:Adding user postgres to group ssl-cert 2024-03-31T15:52:34.297 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:34.297 INFO:teuthology.orchestra.run.smithi090.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-31T15:52:34.541 INFO:teuthology.orchestra.run.smithi102.stdout:syncing data to disk ... ok 2024-03-31T15:52:34.627 INFO:teuthology.orchestra.run.smithi090.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-31T15:52:34.627 INFO:teuthology.orchestra.run.smithi090.stdout:Removing obsolete dictionary files: 2024-03-31T15:52:35.037 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-31T15:52:35.355 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-31T15:52:35.886 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-31T15:52:36.319 INFO:teuthology.orchestra.run.smithi090.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-31T15:52:36.339 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-31T15:52:36.397 INFO:teuthology.orchestra.run.smithi090.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-31T15:52:36.397 INFO:teuthology.orchestra.run.smithi090.stdout:This user must also own the server process. 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:The database cluster will be initialized with locale "en_US". 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:The default text search configuration will be set to "english". 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:Data page checksums are disabled. 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:36.398 INFO:teuthology.orchestra.run.smithi090.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-31T15:52:36.399 INFO:teuthology.orchestra.run.smithi090.stdout:creating subdirectories ... ok 2024-03-31T15:52:36.399 INFO:teuthology.orchestra.run.smithi090.stdout:selecting dynamic shared memory implementation ... posix 2024-03-31T15:52:36.422 INFO:teuthology.orchestra.run.smithi090.stdout:selecting default max_connections ... 100 2024-03-31T15:52:36.455 INFO:teuthology.orchestra.run.smithi090.stdout:selecting default shared_buffers ... 128MB 2024-03-31T15:52:36.469 INFO:teuthology.orchestra.run.smithi090.stdout:selecting default time zone ... Etc/UTC 2024-03-31T15:52:36.469 INFO:teuthology.orchestra.run.smithi090.stdout:creating configuration files ... ok 2024-03-31T15:52:36.632 INFO:teuthology.orchestra.run.smithi090.stdout:running bootstrap script ... ok 2024-03-31T15:52:37.253 INFO:teuthology.orchestra.run.smithi090.stdout:performing post-bootstrap initialization ... ok 2024-03-31T15:52:37.773 INFO:teuthology.orchestra.run.smithi090.stdout:syncing data to disk ... ok 2024-03-31T15:52:38.062 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:38.188 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-31T15:52:38.287 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-31T15:52:38.348 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up postgresql (14+238) ... 2024-03-31T15:52:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:38.558 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:38.792 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:52:38.792 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:52:39.510 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:52:39.628 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-31T15:52:39.963 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:52:40.107 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:52:41.305 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:41.431 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-31T15:52:41.528 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-31T15:52:41.558 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up postgresql (14+238) ... 2024-03-31T15:52:41.628 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:52:41.685 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-31T15:52:41.811 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-31T15:52:42.056 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:52:42.057 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-31T15:52:42.148 INFO:teuthology.orchestra.run.smithi102.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:52:42.156 DEBUG:teuthology.parallel:result is None 2024-03-31T15:52:42.805 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-31T15:52:42.906 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-31T15:52:43.358 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-31T15:52:43.512 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-31T15:52:44.909 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-31T15:52:45.009 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-31T15:52:45.564 INFO:teuthology.orchestra.run.smithi090.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-31T15:52:45.571 DEBUG:teuthology.parallel:result is None 2024-03-31T15:52:45.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:52:45.926 DEBUG:teuthology.orchestra.run.smithi090:> dpkg-query -W -f '${Version}' ceph 2024-03-31T15:52:45.948 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-2574-gd2271579-1jammy 2024-03-31T15:52:45.949 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:52:45.949 INFO:teuthology.task.install:The correct ceph version 19.0.0-2574-gd2271579-1jammy is installed. 2024-03-31T15:52:45.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2271579d5e735cfd6d436b8349c004f06527cb0 2024-03-31T15:52:46.299 DEBUG:teuthology.orchestra.run.smithi102:> dpkg-query -W -f '${Version}' ceph 2024-03-31T15:52:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-2574-gd2271579-1jammy 2024-03-31T15:52:46.322 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2574-gd2271579-1jammy 2024-03-31T15:52:46.322 INFO:teuthology.task.install:The correct ceph version 19.0.0-2574-gd2271579-1jammy is installed. 2024-03-31T15:52:46.324 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-31T15:52:46.324 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:46.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T15:52:46.344 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:46.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-31T15:52:46.386 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-31T15:52:46.387 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:46.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T15:52:46.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T15:52:46.468 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:46.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-31T15:52:46.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-31T15:52:46.553 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-31T15:52:46.554 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:46.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T15:52:46.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T15:52:46.636 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:46.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-31T15:52:46.656 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-31T15:52:46.722 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-31T15:52:46.722 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:46.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T15:52:46.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T15:52:46.799 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:46.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-31T15:52:46.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-31T15:52:46.883 INFO:teuthology.run_tasks:Running task ceph... 2024-03-31T15:52:46.969 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-31T15:52:46.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2024-03-31T15:52:46.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2024-03-31T15:52:46.994 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-31T15:52:46.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-31T15:52:47.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-31T15:52:47.053 INFO:tasks.ceph:Creating extra log directories... 2024-03-31T15:52:47.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-31T15:52:47.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-31T15:52:47.121 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-31T15:52:47.122 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-31T15:52:47.122 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-31T15:52:47.122 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:47.122 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-31T15:52:47.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-31T15:52:47.245 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:47.245 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-31T15:52:47.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-31T15:52:47.350 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-31T15:52:47.350 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-31T15:52:47.350 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664', 'branch': 'wip-vshankar-testing-20240330.170739', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/pool-perm}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7631664', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240330.170739'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240330.170739', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6891, 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'sleep_before_teardown': 0, 'subset': '932/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240330.170739', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKckH9MSjojHRMLXVlBU2myYnNZCRAuBqTRWv9TLz664o+HCyz7XYHhQg/z6xP5TlufTtSBUbEPqjadTUuGjeXM=', 'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBb/CxZGUrMOZXbVtwaG8bt0zun6ADwzwAEZXPWOWexVynz8YWDsB3qZv8Z66DgK2HJsZ1/00hqYcnbT8T33ugM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd2271579d5e735cfd6d436b8349c004f06527cb0', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-30_23:12:08', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-31T15:52:47.351 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-31T15:52:47.354 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-31T15:52:47.362 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2024-03-31T15:52:47.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2024-03-31T15:52:47.429 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:47.429 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T15:52:47.469 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T15:52:47.469 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-03-31T15:52:47.522 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T15:52:47.522 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:47.522 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-03-31T15:52:47.522 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:47.522 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-31 15:46:26.513183564 +0000 2024-03-31T15:52:47.523 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-31 15:46:26.241185990 +0000 2024-03-31T15:52:47.523 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-31 15:46:26.241185990 +0000 2024-03-31T15:52:47.523 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-31 15:46:26.241185990 +0000 2024-03-31T15:52:47.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T15:52:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-31T15:52:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-31T15:52:47.583 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000658021 s, 778 kB/s 2024-03-31T15:52:47.584 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T15:52:47.635 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-31 15:46:26.749181459 +0000 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-31 15:46:26.745181495 +0000 2024-03-31T15:52:47.685 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-31 15:46:26.745181495 +0000 2024-03-31T15:52:47.686 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-31 15:46:26.745181495 +0000 2024-03-31T15:52:47.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T15:52:47.741 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-31T15:52:47.742 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-31T15:52:47.742 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000341239 s, 1.5 MB/s 2024-03-31T15:52:47.743 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T15:52:47.795 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-03-31T15:52:47.845 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-31 15:46:27.517174608 +0000 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-31 15:46:27.249176998 +0000 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-31 15:46:27.249176998 +0000 2024-03-31T15:52:47.846 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-31 15:46:27.249176998 +0000 2024-03-31T15:52:47.846 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T15:52:47.905 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-31T15:52:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-31T15:52:47.906 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000338216 s, 1.5 MB/s 2024-03-31T15:52:47.907 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T15:52:47.958 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-31 15:46:27.785172217 +0000 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-31 15:46:27.781172253 +0000 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-31 15:46:27.781172253 +0000 2024-03-31T15:52:48.009 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-31 15:46:27.781172253 +0000 2024-03-31T15:52:48.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T15:52:48.066 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-31T15:52:48.066 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-31T15:52:48.066 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00034587 s, 1.5 MB/s 2024-03-31T15:52:48.068 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T15:52:48.119 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:48.119 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:48.120 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-31T15:52:48.128 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-31T15:52:48.128 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-31 15:46:19.044964064 +0000 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-31 15:46:18.800966243 +0000 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-31 15:46:18.800966243 +0000 2024-03-31T15:52:48.179 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-31 15:46:18.800966243 +0000 2024-03-31T15:52:48.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-31T15:52:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-31T15:52:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-31T15:52:48.241 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000643819 s, 795 kB/s 2024-03-31T15:52:48.242 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-31T15:52:48.296 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-31T15:52:48.346 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-31T15:52:48.346 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-31 15:46:19.552959527 +0000 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-31 15:46:19.304961742 +0000 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-31 15:46:19.304961742 +0000 2024-03-31T15:52:48.347 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-31 15:46:19.304961742 +0000 2024-03-31T15:52:48.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-31T15:52:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-31T15:52:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-31T15:52:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000368652 s, 1.4 MB/s 2024-03-31T15:52:48.410 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-31T15:52:48.464 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-31T15:52:48.515 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-31T15:52:48.515 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:48.515 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-03-31T15:52:48.515 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:48.516 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-31 15:46:20.056955025 +0000 2024-03-31T15:52:48.516 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-31 15:46:19.804957276 +0000 2024-03-31T15:52:48.516 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-31 15:46:19.804957276 +0000 2024-03-31T15:52:48.516 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-31 15:46:19.804957276 +0000 2024-03-31T15:52:48.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-31T15:52:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-31T15:52:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-31T15:52:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000379721 s, 1.3 MB/s 2024-03-31T15:52:48.578 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-31T15:52:48.633 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 942 Links: 1 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-31 15:46:20.596950201 +0000 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-31 15:46:20.320952666 +0000 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-31 15:46:20.320952666 +0000 2024-03-31T15:52:48.683 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-31 15:46:20.320952666 +0000 2024-03-31T15:52:48.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-31T15:52:48.744 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-31T15:52:48.745 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-31T15:52:48.745 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000381297 s, 1.3 MB/s 2024-03-31T15:52:48.747 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-31T15:52:48.800 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:48.800 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-31T15:52:48.800 INFO:tasks.ceph:Generating config... 2024-03-31T15:52:48.802 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-31T15:52:48.803 INFO:tasks.ceph:[client] debug client = 20 2024-03-31T15:52:48.803 INFO:tasks.ceph:[client] debug ms = 1 2024-03-31T15:52:48.803 INFO:tasks.ceph:[client] pid file = 2024-03-31T15:52:48.803 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-31T15:52:48.803 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-31T15:52:48.803 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-31T15:52:48.804 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-31T15:52:48.805 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-31T15:52:48.806 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-31T15:52:48.806 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-31T15:52:48.806 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-31T15:52:48.807 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-31T15:52:48.808 INFO:tasks.ceph:Setting up mon.a... 2024-03-31T15:52:48.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-31T15:52:48.851 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2024-03-31T15:52:48.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-31T15:52:48.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:52:48.998 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.b', '172.21.15.102'), ('mon.c', '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]')] 2024-03-31T15:52:48.998 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,172.21.15.102,[v2:172.21.15.102:3301,v1:172.21.15.102:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-31T15:52:48.999 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:48.999 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-31T15:52:49.053 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --add b 172.21.15.102 --addv c '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid 0135ca30-606a-448f-b61f-3ca22bf9710a 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = quincy 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 0135ca30-606a-448f-b61f-3ca22bf9710a 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2024-03-31T15:52:49.154600+0000 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:created 2024-03-31T15:52:49.154600+0000 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 17 (quincy) 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 1 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2024-03-31T15:52:49.143 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.b 2024-03-31T15:52:49.144 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon.c 2024-03-31T15:52:49.144 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-31T15:52:49.146 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-31T15:52:49.153 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0135ca30-606a-448f-b61f-3ca22bf9710a... 2024-03-31T15:52:49.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:52:49.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-31T15:52:49.238 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-03-31T15:52:49.239 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-03-31T15:52:49.240 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-03-31T15:52:49.241 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,172.21.15.102,[v2:172.21.15.102:3301,v1:172.21.15.102:6790]" 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 0135ca30-606a-448f-b61f-3ca22bf9710a 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-03-31T15:52:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 20 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 20 2024-03-31T15:52:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:52:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-03-31T15:52:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon down mkfs grace = 300 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: mon op complaint time = 120 2024-03-31T15:52:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:52:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: client mount timeout = 600 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: debug client = 20 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = "" 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-03-31T15:52:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout:[mds] 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds = 20 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: debug mds balancer = 20 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug frag = True 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: mds debug scatterstat = True 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: mds op complaint time = 180 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: mds verify scatter = True 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: osd op complaint time = 180 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: rados mon op timeout = 900 2024-03-31T15:52:49.250 INFO:teuthology.orchestra.run.smithi090.stdout: rados osd op timeout = 900 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2024-03-31T15:52:49.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2024-03-31T15:52:49.252 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.253 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "172.21.15.90,172.21.15.102,[v2:172.21.15.102:3301,v1:172.21.15.102:6790]" 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 0135ca30-606a-448f-b61f-3ca22bf9710a 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock profile = high_recovery_ops 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.254 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug trim objects = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2024-03-31T15:52:49.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 20 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 20 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2024-03-31T15:52:49.256 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-03-31T15:52:49.257 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2024-03-31T15:52:49.258 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2024-03-31T15:52:49.259 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: mon down mkfs grace = 300 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: mon op complaint time = 120 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: client mount timeout = 600 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-03-31T15:52:49.260 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = "" 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout:[mds] 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds = 20 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds balancer = 20 2024-03-31T15:52:49.261 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug frag = True 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug scatterstat = True 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: mds op complaint time = 180 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: mds verify scatter = True 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-31T15:52:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-31T15:52:49.264 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-31T15:52:49.264 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-31T15:52:49.361 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-31T15:52:49.361 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:49.361 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-31T15:52:49.370 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:49.370 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-31T15:52:49.421 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:52:49.421 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:49.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-31T15:52:49.421 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:52:49.492 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:52:49.492 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:49.545 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:52:49.546 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:49.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-31T15:52:49.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-31T15:52:49.579 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:52:49.579 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:49.630 INFO:tasks.ceph:Setting up mon nodes... 2024-03-31T15:52:49.630 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-31T15:52:49.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-31T15:52:49.685 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-31T15:52:49.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-31T15:52:49.747 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-31T15:52:49.767 INFO:tasks.ceph:Setting up mds nodes... 2024-03-31T15:52:49.768 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-31T15:52:49.821 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-31T15:52:49.825 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-31T15:52:49.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-31T15:52:49.938 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-31T15:52:49.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-31T15:52:49.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-31T15:52:50.020 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-31T15:52:50.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-31T15:52:50.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-31T15:52:50.139 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-31T15:52:50.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-31T15:52:50.163 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-31T15:52:50.163 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-31T15:52:50.205 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-31T15:52:50.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-31T15:52:50.305 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-31T15:52:50.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-31T15:52:50.364 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-31T15:52:50.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-31T15:52:50.466 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-31T15:52:50.483 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-31T15:52:50.483 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-31T15:52:50.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-31T15:52:50.502 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:50.502 INFO:tasks.ceph:role: osd.0 2024-03-31T15:52:50.502 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:52:50.502 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-31T15:52:50.571 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:52:50.572 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:52:51.060 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-31T15:52:51.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-31T15:52:51.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-31T15:52:51.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-31T15:52:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:52:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:52:51.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-31T15:52:51.292 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:51.292 INFO:tasks.ceph:role: osd.1 2024-03-31T15:52:51.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:52:51.293 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:52:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:52:51.800 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-31T15:52:51.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-31T15:52:51.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-31T15:52:51.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-31T15:52:51.921 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:52:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:52:51.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-31T15:52:51.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:51.989 INFO:tasks.ceph:role: osd.2 2024-03-31T15:52:51.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:52:51.990 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-31T15:52:52.060 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:52:52.061 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:52:52.528 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-31T15:52:52.531 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-31T15:52:52.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-31T15:52:52.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-31T15:52:52.690 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:52:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:52:52.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-31T15:52:52.757 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-31T15:52:52.757 INFO:tasks.ceph:role: osd.3 2024-03-31T15:52:52.757 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:52:52.757 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-31T15:52:52.825 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:52:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:52:53.314 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-31T15:52:53.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-31T15:52:53.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-31T15:52:53.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-31T15:52:53.514 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:52:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:52:53.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:53.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.661+0000 7f29fd4487c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-31T15:52:53.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.665+0000 7f29fd4487c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-31T15:52:53.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.665+0000 7f29fd4487c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:52:53.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.665+0000 7f29fd4487c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:52:53.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.665+0000 7f29fd4487c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-31T15:52:53.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:53.669+0000 7f29fd4487c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-31T15:52:56.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-31T15:52:56.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:56.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.781+0000 7f3470f387c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-31T15:52:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.781+0000 7f3470f387c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-31T15:52:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.781+0000 7f3470f387c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:52:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.781+0000 7f3470f387c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:52:56.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.781+0000 7f3470f387c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-31T15:52:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:56.785+0000 7f3470f387c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-31T15:52:59.756 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-31T15:52:59.776 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:52:59.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-31T15:52:59.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-31T15:52:59.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:52:59.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:52:59.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-31T15:52:59.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:52:59.881+0000 7f740330b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-31T15:53:02.871 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-31T15:53:02.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:02.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.989+0000 7fe3f54ba7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-31T15:53:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.993+0000 7fe3f54ba7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-31T15:53:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.993+0000 7fe3f54ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:53:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.993+0000 7fe3f54ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:53:02.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.993+0000 7fe3f54ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-31T15:53:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:02.993+0000 7fe3f54ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-31T15:53:05.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-31T15:53:05.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-31T15:53:05.984 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:53:05.984 INFO:tasks.ceph:role: osd.4 2024-03-31T15:53:05.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:53:05.985 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-31T15:53:06.053 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:53:06.053 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:53:06.053 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:53:06.053 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:53:06.053 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:53:06.054 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:53:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:53:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:53:06.054 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:53:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:53:06.508 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-31T15:53:06.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-31T15:53:06.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-31T15:53:06.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-31T15:53:06.632 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:53:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:53:06.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-31T15:53:06.693 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:53:06.693 INFO:tasks.ceph:role: osd.5 2024-03-31T15:53:06.693 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:53:06.693 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-31T15:53:06.760 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:53:06.761 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:53:07.230 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-31T15:53:07.233 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-31T15:53:07.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-31T15:53:07.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-31T15:53:07.412 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:53:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:53:07.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-31T15:53:07.474 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:53:07.474 INFO:tasks.ceph:role: osd.6 2024-03-31T15:53:07.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:53:07.474 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-31T15:53:07.545 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:53:07.545 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:53:07.545 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:53:07.546 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:53:08.018 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-31T15:53:08.021 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-31T15:53:08.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-31T15:53:08.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-31T15:53:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:53:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:53:08.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-31T15:53:08.246 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-31T15:53:08.246 INFO:tasks.ceph:role: osd.7 2024-03-31T15:53:08.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:53:08.246 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-31T15:53:08.317 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-31T15:53:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-31T15:53:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-31T15:53:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-31T15:53:08.317 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-31T15:53:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-31T15:53:08.318 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-31T15:53:08.318 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-31T15:53:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-31T15:53:08.318 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-31T15:53:08.762 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-31T15:53:08.765 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-31T15:53:08.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-31T15:53:08.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-31T15:53:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: /sbin/restorecon: command not found 2024-03-31T15:53:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:53:08.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:09.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.029+0000 7f71f72957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-31T15:53:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.029+0000 7f71f72957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-31T15:53:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.029+0000 7f71f72957c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:53:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.029+0000 7f71f72957c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:53:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.029+0000 7f71f72957c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-31T15:53:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:09.037+0000 7f71f72957c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-31T15:53:11.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-31T15:53:12.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:12.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.121+0000 7f823502b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-31T15:53:12.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.125+0000 7f823502b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-31T15:53:12.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.125+0000 7f823502b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:53:12.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.125+0000 7f823502b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:53:12.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.125+0000 7f823502b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-31T15:53:12.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:12.125+0000 7f823502b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-31T15:53:15.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-31T15:53:15.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.209+0000 7f98f0ba97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-31T15:53:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.213+0000 7f98f0ba97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-31T15:53:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.213+0000 7f98f0ba97c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:53:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.213+0000 7f98f0ba97c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:53:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.213+0000 7f98f0ba97c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-31T15:53:15.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:15.213+0000 7f98f0ba97c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-31T15:53:17.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:53:17.356 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:53:18.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-31T15:53:18.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:18.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.285+0000 7fb7dd9d37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-31T15:53:18.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.285+0000 7fb7dd9d37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-31T15:53:18.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.285+0000 7fb7dd9d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:53:18.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.285+0000 7fb7dd9d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:53:18.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.285+0000 7fb7dd9d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-31T15:53:18.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:18.289+0000 7fb7dd9d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-31T15:53:21.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-31T15:53:21.265 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-31T15:53:21.265 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-31T15:53:21.287 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.287 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-31T15:53:21.350 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-31T15:53:21.414 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-31T15:53:21.477 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.477 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-31T15:53:21.536 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.536 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-31T15:53:21.596 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:21.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-31T15:53:21.656 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-31T15:53:21.677 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.677 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-31T15:53:21.737 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-31T15:53:21.797 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-31T15:53:21.857 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-31T15:53:21.917 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-31T15:53:21.977 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:21.978 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-31T15:53:22.037 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:22.037 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-31T15:53:22.046 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:22.046 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-31T15:53:22.098 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:22.098 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-31T15:53:22.106 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:22.106 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-31T15:53:22.158 INFO:tasks.ceph:Adding keys to all mons... 2024-03-31T15:53:22.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-31T15:53:22.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-31T15:53:22.205 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2024-03-31T15:53:22.205 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDRhglmy8KFKRAAyHCJHySxd3Jcb8lQBIA7eA== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.a] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDRhglmcninMRAARhope6BpTuawcm/Bd/MSIg== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.c] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDRhglmcqabOBAAqFr5SZkMULMCOz06c7V0qw== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDVhglm8T+UJxAA3z+5rswh0d7RakAvfO7J8g== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDYhglmke27LhAAF0oYalYC+cMjqc4Iw1r4kg== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDbhglmcDWmNBAAuAJakIes6OBv3yzAhtT0DA== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDehglm7Y42OxAA8+5GF4XF+3cTnxhi2BhiRw== 2024-03-31T15:53:22.206 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDRhglmdRVMLRAAHrLHygDnOeGTKMmMCVyzpg== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.b] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmli3zARAAHVKo3x7gWq9M18P3MNqxXA== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[mds.d] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglm+6ITCRAACtqVZ3FpKoRXagsrFx3ywg== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDlhglmJwDhARAAzCpZr8/syvLjfnI0qi4jpA== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDohglmGXmMBxAAQKoVfHNdxRWZIjtF3TN8CA== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDrhglmjsnMDBAAYOGRa9UwgXIIjj6hbg8fwg== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDuhglmcWRFERAAWDwc/7Xlku2oXqBELqBCPg== 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:53:22.207 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout:[client.2] 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmDJHvEhAA/EJD8+c4iK8u7ZEroAHp9g== 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmSpJzFhAAdwvdZGC6Tfafcb5kqX9/Zw== 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout:[client.3] 2024-03-31T15:53:22.208 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglm2+uRHBAABJCOXHPmeC+WUichEcsMHw== 2024-03-31T15:53:22.219 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2024-03-31T15:53:22.219 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDRhglmy8KFKRAAyHCJHySxd3Jcb8lQBIA7eA== 2024-03-31T15:53:22.219 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-31T15:53:22.219 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDRhglmcninMRAARhope6BpTuawcm/Bd/MSIg== 2024-03-31T15:53:22.219 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDRhglmcqabOBAAqFr5SZkMULMCOz06c7V0qw== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDVhglm8T+UJxAA3z+5rswh0d7RakAvfO7J8g== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDYhglmke27LhAAF0oYalYC+cMjqc4Iw1r4kg== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDbhglmcDWmNBAAuAJakIes6OBv3yzAhtT0DA== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDehglm7Y42OxAA8+5GF4XF+3cTnxhi2BhiRw== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDRhglmdRVMLRAAHrLHygDnOeGTKMmMCVyzpg== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglmli3zARAAHVKo3x7gWq9M18P3MNqxXA== 2024-03-31T15:53:22.220 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglm+6ITCRAACtqVZ3FpKoRXagsrFx3ywg== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDlhglmJwDhARAAzCpZr8/syvLjfnI0qi4jpA== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDohglmGXmMBxAAQKoVfHNdxRWZIjtF3TN8CA== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.6] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDrhglmjsnMDBAAYOGRa9UwgXIIjj6hbg8fwg== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.7] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDuhglmcWRFERAAWDwc/7Xlku2oXqBELqBCPg== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout:[client.2] 2024-03-31T15:53:22.221 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglmDJHvEhAA/EJD8+c4iK8u7ZEroAHp9g== 2024-03-31T15:53:22.222 INFO:teuthology.orchestra.run.smithi102.stdout:[client.1] 2024-03-31T15:53:22.222 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglmSpJzFhAAdwvdZGC6Tfafcb5kqX9/Zw== 2024-03-31T15:53:22.222 INFO:teuthology.orchestra.run.smithi102.stdout:[client.3] 2024-03-31T15:53:22.222 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDShglm2+uRHBAABJCOXHPmeC+WUichEcsMHw== 2024-03-31T15:53:22.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:53:22.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:53:22.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.383 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.439 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.663 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:53:22.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-31T15:53:22.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-31T15:53:22.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:22.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:23.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:23.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-31T15:53:23.057 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.116 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-31T15:53:23.293 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-31T15:53:23.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-31T15:53:23.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:53:23.417 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-31T15:53:23.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-31T15:53:23.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:53:23.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-31T15:53:23.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-31T15:53:23.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-31T15:53:23.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-31T15:53:23.776 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:23.779 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-31T15:53:23.825 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-31T15:53:23.825 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-31T15:53:23.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-31T15:53:23.829 INFO:tasks.ceph.mon.a:Started 2024-03-31T15:53:23.829 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-31T15:53:23.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-31T15:53:23.873 INFO:tasks.ceph.mon.b:Started 2024-03-31T15:53:23.873 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-31T15:53:23.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-31T15:53:23.876 INFO:tasks.ceph.mon.c:Started 2024-03-31T15:53:23.876 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-31T15:53:23.876 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-31T15:53:23.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-31T15:53:23.879 INFO:tasks.ceph.mgr.y:Started 2024-03-31T15:53:23.880 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-31T15:53:23.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-31T15:53:23.882 INFO:tasks.ceph.mgr.x:Started 2024-03-31T15:53:23.883 INFO:tasks.ceph:Setting crush tunables to default 2024-03-31T15:53:23.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2024-03-31T15:53:24.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 Processor -- start 2024-03-31T15:53:24.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 -- start start 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 0x7fc2f8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80b2d30 con 0x7fc2f815bbb0 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f80b3010 con 0x7fc2f80b4c20 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc300600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80b32f0 con 0x7fc2f815bf90 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fdc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc2f815bbb0 0x7fc2f815f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59834/0 (socket says 172.21.15.90:59834) 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41158/0 (socket says 172.21.15.90:41158) 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fdc00640 1 -- 172.21.15.90:0/2205209811 learned_addr learned my addr 172.21.15.90:0/2205209811 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fd200640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 0x7fc2f8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 msgr2=0x7fc2f80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-31T15:53:24.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 msgr2=0x7fc2f80717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fd200640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 msgr2=0x7fc2f8071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fd200640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 msgr2=0x7fc2f8071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fd200640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 0x7fc2f8071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-31T15:53:24.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.028+0000 7fc2fd200640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 0x7fc2f8071260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-31T15:53:24.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159671539 0 0) 0x7fc2f80b2d30 con 0x7fc2f815bbb0 2024-03-31T15:53:24.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2e0003530 con 0x7fc2f815bbb0 2024-03-31T15:53:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4163654720 0 0) 0x7fc2e0003530 con 0x7fc2f815bbb0 2024-03-31T15:53:24.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 msgr2=0x7fc2f8071260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f815bf90 0x7fc2f8071260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 msgr2=0x7fc2f80717a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 --2- 172.21.15.90:0/2205209811 >> v2:172.21.15.90:3300/0 conn(0x7fc2f80b4c20 0x7fc2f80717a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f80b35d0 con 0x7fc2f815bbb0 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fc2ec001680 con 0x7fc2f815bbb0 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 >> v1:172.21.15.90:6789/0 conn(0x7fc2f815bbb0 legacy=0x7fc2f815f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e00039c0 0x7fc2e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2e0004810 0x7fc2e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80b3010 con 0x7fc2e0005190 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80b32f0 con 0x7fc2e00039c0 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e000bf10 con 0x7fc2e0004810 2024-03-31T15:53:24.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.184+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fdc00640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e00039c0 0x7fc2e0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fd200640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2e0004810 0x7fc2e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2e0004810 msgr2=0x7fc2e0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2e0004810 0x7fc2e0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e00039c0 msgr2=0x7fc2e0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2e00039c0 0x7fc2e0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e0003530 con 0x7fc2e0005190 2024-03-31T15:53:24.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.188+0000 7fc2fe600640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2e000b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009610 tx=0x7fc2f001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bf6f469547e6df server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00b2070 con 0x7fc2e0005190 2024-03-31T15:53:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00a5d20 con 0x7fc2e0005190 2024-03-31T15:53:24.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc2f7e00640 1 -- 172.21.15.90:0/2205209811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0009bd0 con 0x7fc2e0005190 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- 172.21.15.90:0/2205209811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 msgr2=0x7fc2e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 --2- 172.21.15.90:0/2205209811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2e000b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009610 tx=0x7fc2f001ce20 comp rx=0 tx=0).stop 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc2fe600640 1 -- 172.21.15.90:0/2205209811 reap_dead start 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- 172.21.15.90:0/2205209811 shutdown_connections 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- 172.21.15.90:0/2205209811 >> 172.21.15.90:0/2205209811 conn(0x7fc2f80b18b0 msgr2=0x7fc2f80b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- 172.21.15.90:0/2205209811 shutdown_connections 2024-03-31T15:53:24.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- 172.21.15.90:0/2205209811 wait complete. 2024-03-31T15:53:24.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 Processor -- start 2024-03-31T15:53:24.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.192+0000 7fc300600640 1 -- start start 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f81f7610 0x7fc2f81f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f81f7fd0 0x7fc2f81e69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80b3f50 con 0x7fc2e0005190 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80b3dd0 con 0x7fc2f81f7610 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80aefa0 con 0x7fc2f81f7fd0 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41200/0 (socket says 172.21.15.90:41200) 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 -- 172.21.15.90:0/3085171186 learned_addr learned my addr 172.21.15.90:0/3085171186 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fe600640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f81f7fd0 0x7fc2f81e69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fd200640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f81f7610 0x7fc2f81f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 -- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f81f7fd0 msgr2=0x7fc2f81e69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f81f7fd0 0x7fc2f81e69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 -- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f81f7610 msgr2=0x7fc2f81f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f81f7610 0x7fc2f81f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 -- 172.21.15.90:0/3085171186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f81f8480 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2fdc00640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007310 tx=0x7fc2ec0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00f9c0 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec018070 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f8155690 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00fde0 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f8155c50 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2ec01c7a0 con 0x7fc2e0005190 2024-03-31T15:53:24.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc2ec013c20 con 0x7fc2e0005190 2024-03-31T15:53:24.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.196+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f8075880 con 0x7fc2e0005190 2024-03-31T15:53:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.208+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc2ec013e20 con 0x7fc2e0005190 2024-03-31T15:53:24.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.268+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc2f80685f0 con 0x7fc2e0005190 2024-03-31T15:53:24.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.272+0000 7fc2f5600640 1 -- 172.21.15.90:0/3085171186 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fc2ec0133f0 con 0x7fc2e0005190 2024-03-31T15:53:24.265 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-03-31T15:53:24.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 msgr2=0x7fc2f81e83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec007310 tx=0x7fc2ec0077d0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 shutdown_connections 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f81f7fd0 0x7fc2f81e69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f81f7610 0x7fc2f81f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 --2- 172.21.15.90:0/3085171186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2e0005190 0x7fc2f81e83c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 >> 172.21.15.90:0/3085171186 conn(0x7fc2f80b18b0 msgr2=0x7fc2f815a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 shutdown_connections 2024-03-31T15:53:24.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:24.276+0000 7fc300600640 1 -- 172.21.15.90:0/3085171186 wait complete. 2024-03-31T15:53:24.285 INFO:tasks.ceph:check_enable_crimson: False 2024-03-31T15:53:24.285 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-31T15:53:24.285 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:24.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-31T15:53:24.305 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:24.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-31T15:53:24.369 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:24.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-31T15:53:24.381 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:24.388+0000 7f60dc64b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T15:53:24.386 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:24.393+0000 7f7df684c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-31T15:53:24.401 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:24.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-31T15:53:24.462 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:24.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-31T15:53:24.481 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:24.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-31T15:53:24.541 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:24.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-31T15:53:24.601 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:53:24.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-31T15:53:24.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new c67e90a6-0b88-4027-a912-d44a80e48d1d 0 2024-03-31T15:53:24.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 Processor -- start 2024-03-31T15:53:24.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 -- start start 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f85fc162400 0x7f85fc162890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85fc072de0 con 0x7f85fc0ccb30 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f85fc162dd0 con 0x7f85fc162400 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f8602600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f85fc0730c0 con 0x7f85fc06b820 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37044/0 (socket says 172.21.15.102:37044) 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 -- 172.21.15.102:0/3289330131 learned_addr learned my addr 172.21.15.102:0/3289330131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85fbe00640 1 --2- 172.21.15.102:0/3289330131 >> v2:172.21.15.90:3300/0 conn(0x7f85fc162400 0x7f85fc162890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 -- 172.21.15.102:0/3289330131 >> v2:172.21.15.90:3300/0 conn(0x7f85fc162400 msgr2=0x7f85fc162890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 --2- 172.21.15.102:0/3289330131 >> v2:172.21.15.90:3300/0 conn(0x7f85fc162400 0x7f85fc162890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 -- 172.21.15.102:0/3289330131 >> v1:172.21.15.90:6789/0 conn(0x7f85fc0ccb30 legacy=0x7f85fc06e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.849+0000 7f85faa00640 1 -- 172.21.15.102:0/3289330131 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc0733a0 con 0x7f85fc06b820 2024-03-31T15:53:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f85faa00640 1 --2- 172.21.15.102:0/3289330131 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009870 tx=0x7f85e801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb7ad5bb4d1b4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f85fa000640 1 -- 172.21.15.102:0/3289330131 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80b1070 con 0x7f85fc06b820 2024-03-31T15:53:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f85fa000640 1 -- 172.21.15.102:0/3289330131 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e80a8050 con 0x7f85fc06b820 2024-03-31T15:53:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f85fa000640 1 -- 172.21.15.102:0/3289330131 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80ac860 con 0x7f85fc06b820 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 -- 172.21.15.102:0/3289330131 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 msgr2=0x7f85fc06bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 --2- 172.21.15.102:0/3289330131 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009870 tx=0x7f85e801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 -- 172.21.15.102:0/3289330131 shutdown_connections 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 --2- 172.21.15.102:0/3289330131 >> v2:172.21.15.90:3300/0 conn(0x7f85fc162400 0x7f85fc162890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 --2- 172.21.15.102:0/3289330131 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc06bc80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 -- 172.21.15.102:0/3289330131 >> 172.21.15.102:0/3289330131 conn(0x7f85fc0b29a0 msgr2=0x7f85fc0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 -- 172.21.15.102:0/3289330131 shutdown_connections 2024-03-31T15:53:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.853+0000 7f8602600640 1 -- 172.21.15.102:0/3289330131 wait complete. 2024-03-31T15:53:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 Processor -- start 2024-03-31T15:53:24.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- start start 2024-03-31T15:53:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 0x7f85fc154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85fc162400 0x7f85fc154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc072ba0 con 0x7f85fc162400 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc072a20 con 0x7f85fc0ccb30 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f85fc0d0900 con 0x7f85fc06b820 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85fb400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85fb400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37046/0 (socket says 172.21.15.102:37046) 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 0x7f85fc154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85fb400640 1 -- 172.21.15.102:0/227646207 learned_addr learned my addr 172.21.15.102:0/227646207 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85fbe00640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85fc162400 0x7f85fc154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 -- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 msgr2=0x7f85fc153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 -- 172.21.15.102:0/227646207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85fc162400 msgr2=0x7f85fc154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85fc162400 0x7f85fc154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 -- 172.21.15.102:0/227646207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc15aa80 con 0x7f85fc0ccb30 2024-03-31T15:53:24.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85faa00640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 0x7f85fc154420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85e80ac000 tx=0x7f85e801ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:24.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80b1040 con 0x7f85fc0ccb30 2024-03-31T15:53:24.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e80acc90 con 0x7f85fc0ccb30 2024-03-31T15:53:24.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80b7660 con 0x7f85fc0ccb30 2024-03-31T15:53:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85fc15ad60 con 0x7f85fc0ccb30 2024-03-31T15:53:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.857+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85fc15b320 con 0x7f85fc0ccb30 2024-03-31T15:53:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.861+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f85e80bf030 con 0x7f85fc0ccb30 2024-03-31T15:53:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.861+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f85e80c1c30 con 0x7f85fc0ccb30 2024-03-31T15:53:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.861+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85fc068d00 con 0x7f85fc0ccb30 2024-03-31T15:53:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.873+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f85e80c6020 con 0x7f85fc0ccb30 2024-03-31T15:53:24.888 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:24.896+0000 7f60dc64b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T15:53:24.898 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:24.909+0000 7f7df684c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-31T15:53:24.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.937+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c67e90a6-0b88-4027-a912-d44a80e48d1d", "id": 0} v 0) -- 0x7f85fc072800 con 0x7f85fc0ccb30 2024-03-31T15:53:24.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f85f3400640 1 -- 172.21.15.102:0/227646207 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c67e90a6-0b88-4027-a912-d44a80e48d1d", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f85e8016150 con 0x7f85fc0ccb30 2024-03-31T15:53:24.936 INFO:teuthology.orchestra.run.smithi102.stdout:0 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 msgr2=0x7f85fc154420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 0x7f85fc154420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85e80ac000 tx=0x7f85e801ad10 comp rx=0 tx=0).stop 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 shutdown_connections 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85fc162400 0x7f85fc154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85fc0ccb30 0x7f85fc154420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 --2- 172.21.15.102:0/227646207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f85fc06b820 0x7f85fc153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 >> 172.21.15.102:0/227646207 conn(0x7f85fc0b29a0 msgr2=0x7f85fc0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 shutdown_connections 2024-03-31T15:53:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:24.945+0000 7f8602600640 1 -- 172.21.15.102:0/227646207 wait complete. 2024-03-31T15:53:24.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new e719019d-e1d1-415e-b65f-56fd94db3bae 1 2024-03-31T15:53:25.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 Processor -- start 2024-03-31T15:53:25.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 -- start start 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 0x7f2f9c0cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f9c0cd700 con 0x7f2f9c15fc70 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f9c0cf0a0 con 0x7f2f9c13f080 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2fa2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c0cf380 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 0x7f2f9c0cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2f9b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2f9c13f080 0x7f2f9c06b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51552/0 (socket says 172.21.15.102:51552) 2024-03-31T15:53:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.097+0000 7f2f9b400640 1 -- 172.21.15.102:0/1651431817 learned_addr learned my addr 172.21.15.102:0/1651431817 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-31T15:53:25.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.101+0000 7f2f93e00640 1 -- 172.21.15.102:0/1651431817 >> v1:172.21.15.102:6789/0 conn(0x7f2f9c13f080 legacy=0x7f2f9c06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.101+0000 7f2f93e00640 1 -- 172.21.15.102:0/1651431817 >> v1:172.21.15.90:6789/0 conn(0x7f2f9c15fc70 legacy=0x7f2f9c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.101+0000 7f2f93e00640 1 -- 172.21.15.102:0/1651431817 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c0cf660 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.101+0000 7f2f9aa00640 1 -- 172.21.15.102:0/1651431817 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554092625 0 0) 0x7f2f9c0cf0a0 con 0x7f2f9c13f080 2024-03-31T15:53:25.093 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.100+0000 7f60dc64b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T15:53:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2f93e00640 1 --2- 172.21.15.102:0/1651431817 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 0x7f2f9c0cffc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f840097f0 tx=0x7f2f8401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21b905e16da990e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2f9aa00640 1 -- 172.21.15.102:0/1651431817 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840b0070 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2f9aa00640 1 -- 172.21.15.102:0/1651431817 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f840ab600 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2f9aa00640 1 -- 172.21.15.102:0/1651431817 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840ab9d0 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- 172.21.15.102:0/1651431817 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 msgr2=0x7f2f9c0cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 --2- 172.21.15.102:0/1651431817 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 0x7f2f9c0cffc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f840097f0 tx=0x7f2f8401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- 172.21.15.102:0/1651431817 shutdown_connections 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 --2- 172.21.15.102:0/1651431817 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c0cfb80 0x7f2f9c0cffc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- 172.21.15.102:0/1651431817 >> 172.21.15.102:0/1651431817 conn(0x7f2f9c0b2950 msgr2=0x7f2f9c0d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- 172.21.15.102:0/1651431817 shutdown_connections 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- 172.21.15.102:0/1651431817 wait complete. 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 Processor -- start 2024-03-31T15:53:25.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.105+0000 7f2fa2600640 1 -- start start 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 0x7f2f9c15d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f9c15fc70 0x7f2f9c137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c13f5e0 con 0x7f2f9c15fc70 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c13f460 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c16a830 con 0x7f2f9c13f080 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 0x7f2f9c15d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46954/0 (socket says 172.21.15.102:46954) 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 -- 172.21.15.102:0/2551808292 learned_addr learned my addr 172.21.15.102:0/2551808292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f9c15fc70 0x7f2f9c137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 -- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 msgr2=0x7f2f9c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 0x7f2f9c15d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 -- 172.21.15.102:0/2551808292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f9c15fc70 msgr2=0x7f2f9c137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f9c15fc70 0x7f2f9c137c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 -- 172.21.15.102:0/2551808292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c0b0220 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93e00640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 0x7f2f9c15d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c4d0 tx=0x7f2f8c00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c018070 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0074e0 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c13d590 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c004a30 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c13db50 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2f8c01b070 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f2f8c0127b0 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.109+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9c068d00 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.113+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2f8c004bd0 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.124 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:25.133+0000 7f7df684c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-31T15:53:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.177+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e719019d-e1d1-415e-b65f-56fd94db3bae", "id": 1} v 0) -- 0x7f2f9c0d1300 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.185+0000 7f2f93400640 1 -- 172.21.15.102:0/2551808292 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e719019d-e1d1-415e-b65f-56fd94db3bae", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f2f8c01d440 con 0x7f2f9c0cfb80 2024-03-31T15:53:25.179 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 msgr2=0x7f2f9c139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00c4d0 tx=0x7f2f8c00c990 comp rx=0 tx=0).stop 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 shutdown_connections 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2f9c15fc70 0x7f2f9c137c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2f9c13f080 0x7f2f9c15d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 --2- 172.21.15.102:0/2551808292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c0cfb80 0x7f2f9c139370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 >> 172.21.15.102:0/2551808292 conn(0x7f2f9c0b2950 msgr2=0x7f2f9c13ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 shutdown_connections 2024-03-31T15:53:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.189+0000 7f2fa2600640 1 -- 172.21.15.102:0/2551808292 wait complete. 2024-03-31T15:53:25.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 1f39c8a8-92d5-4c1e-ba1a-68be683b063c 2 2024-03-31T15:53:25.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 Processor -- start 2024-03-31T15:53:25.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 -- start start 2024-03-31T15:53:25.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7faca806c700 0x7faca80b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faca80afe80 con 0x7faca806dd90 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faca80b3ce0 con 0x7faca806d9b0 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facaf800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7faca80b3fc0 con 0x7faca806c700 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facace00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7faca806d9b0 0x7faca80b4d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51558/0 (socket says 172.21.15.102:51558) 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7facace00640 1 -- 172.21.15.102:0/3978581060 learned_addr learned my addr 172.21.15.102:0/3978581060 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-31T15:53:25.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7e00640 1 --2- 172.21.15.102:0/3978581060 >> v2:172.21.15.90:3300/0 conn(0x7faca806c700 0x7faca80b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686580071 0 0) 0x7faca80b3ce0 con 0x7faca806d9b0 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac90003530 con 0x7faca806d9b0 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7e00640 1 -- 172.21.15.102:0/3978581060 >> v1:172.21.15.102:6789/0 conn(0x7faca806d9b0 legacy=0x7faca80b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7e00640 1 -- 172.21.15.102:0/3978581060 >> v1:172.21.15.90:6789/0 conn(0x7faca806dd90 legacy=0x7faca806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713771979 0 0) 0x7faca80afe80 con 0x7faca806dd90 2024-03-31T15:53:25.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.345+0000 7faca7e00640 1 -- 172.21.15.102:0/3978581060 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca80b42a0 con 0x7faca806c700 2024-03-31T15:53:25.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7faca7e00640 1 --2- 172.21.15.102:0/3978581060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806c700 0x7faca80b3650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac940097c0 tx=0x7fac9401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40e42ce5852c7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac940a7070 con 0x7faca806c700 2024-03-31T15:53:25.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac940b0070 con 0x7faca806c700 2024-03-31T15:53:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7faca7400640 1 -- 172.21.15.102:0/3978581060 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac940ab890 con 0x7faca806c700 2024-03-31T15:53:25.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 -- 172.21.15.102:0/3978581060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806c700 msgr2=0x7faca80b3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 --2- 172.21.15.102:0/3978581060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806c700 0x7faca80b3650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac940097c0 tx=0x7fac9401c470 comp rx=0 tx=0).stop 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 -- 172.21.15.102:0/3978581060 shutdown_connections 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 --2- 172.21.15.102:0/3978581060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806c700 0x7faca80b3650 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 -- 172.21.15.102:0/3978581060 >> 172.21.15.102:0/3978581060 conn(0x7faca80b2950 msgr2=0x7faca80d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 -- 172.21.15.102:0/3978581060 shutdown_connections 2024-03-31T15:53:25.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.349+0000 7facaf800640 1 -- 172.21.15.102:0/3978581060 wait complete. 2024-03-31T15:53:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 Processor -- start 2024-03-31T15:53:25.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 -- start start 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 0x7faca8168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806dd90 0x7faca8150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faca815b4c0 con 0x7faca806dd90 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faca815b340 con 0x7faca806d9b0 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7faca80ce5f0 con 0x7faca806c700 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facace00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 0x7faca8168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46956/0 (socket says 172.21.15.102:46956) 2024-03-31T15:53:25.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 -- 172.21.15.102:0/1198893229 learned_addr learned my addr 172.21.15.102:0/1198893229 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facad800640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806dd90 0x7faca8150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 -- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 msgr2=0x7faca8168ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 0x7faca8168ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 -- 172.21.15.102:0/1198893229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806dd90 msgr2=0x7faca8150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806dd90 0x7faca8150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 -- 172.21.15.102:0/1198893229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca814b510 con 0x7faca806d9b0 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facace00640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 0x7faca8168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:25.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca7e00640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fac940097c0 tx=0x7fac9401acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94004220 con 0x7faca806d9b0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac940043c0 con 0x7faca806d9b0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94003860 con 0x7faca806d9b0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.353+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca814a470 con 0x7faca806d9b0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.357+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca814aa30 con 0x7faca806d9b0 2024-03-31T15:53:25.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.357+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fac940bd070 con 0x7faca806d9b0 2024-03-31T15:53:25.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.357+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fac94003e20 con 0x7faca806d9b0 2024-03-31T15:53:25.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.357+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca8068d00 con 0x7faca806d9b0 2024-03-31T15:53:25.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.361+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fac94016090 con 0x7faca806d9b0 2024-03-31T15:53:25.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.425+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1f39c8a8-92d5-4c1e-ba1a-68be683b063c", "id": 2} v 0) -- 0x7faca80cf6c0 con 0x7faca806d9b0 2024-03-31T15:53:25.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7faca4c00640 1 -- 172.21.15.102:0/1198893229 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f39c8a8-92d5-4c1e-ba1a-68be683b063c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fac94003a00 con 0x7faca806d9b0 2024-03-31T15:53:25.424 INFO:teuthology.orchestra.run.smithi102.stdout:2 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 msgr2=0x7faca816a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fac940097c0 tx=0x7fac9401acb0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 shutdown_connections 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faca806dd90 0x7faca8150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faca806d9b0 0x7faca816a1e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 --2- 172.21.15.102:0/1198893229 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faca806c700 0x7faca8168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 >> 172.21.15.102:0/1198893229 conn(0x7faca80b2950 msgr2=0x7faca80ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 shutdown_connections 2024-03-31T15:53:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.433+0000 7facaf800640 1 -- 172.21.15.102:0/1198893229 wait complete. 2024-03-31T15:53:25.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new ff0b59be-f0c8-44ea-82f1-5b644069371b 3 2024-03-31T15:53:25.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.584+0000 7fb0cc800640 1 Processor -- start 2024-03-31T15:53:25.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.584+0000 7fb0cc800640 1 -- start start 2024-03-31T15:53:25.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0cc800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0c415e4e0 0x7fb0c406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0cc800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0cc800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0c415e8c0 con 0x7fb0c406ce50 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0cc800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb0c40d0ff0 con 0x7fb0c415e4e0 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0cc800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c40d12d0 con 0x7fb0c40cdf20 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0ca800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0c415e4e0 0x7fb0c406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0ca800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37082/0 (socket says 172.21.15.102:37082) 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0ca800640 1 -- 172.21.15.102:0/4066700630 learned_addr learned my addr 172.21.15.102:0/4066700630 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:25.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9400640 1 --1- 172.21.15.102:0/4066700630 >> v1:172.21.15.102:6789/0 conn(0x7fb0c406ce50 0x7fb0c40ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51570/0 (socket says 172.21.15.102:51570) 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9e00640 1 -- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 msgr2=0x7fb0c406d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9e00640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9e00640 1 -- 172.21.15.102:0/4066700630 >> v1:172.21.15.102:6789/0 conn(0x7fb0c406ce50 legacy=0x7fb0c40ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c8a00640 1 -- 172.21.15.102:0/4066700630 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632222606 0 0) 0x7fb0c415e8c0 con 0x7fb0c406ce50 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0c9e00640 1 -- 172.21.15.102:0/4066700630 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c40cc710 con 0x7fb0c415e4e0 2024-03-31T15:53:25.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.588+0000 7fb0ca800640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0c9e00640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c415e4e0 0x7fb0c406c910 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80097f0 tx=0x7fb0b801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b027262673d97c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0c8a00640 1 -- 172.21.15.102:0/4066700630 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b0070 con 0x7fb0c415e4e0 2024-03-31T15:53:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0c8a00640 1 -- 172.21.15.102:0/4066700630 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b80ab600 con 0x7fb0c415e4e0 2024-03-31T15:53:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0c8a00640 1 -- 172.21.15.102:0/4066700630 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80ab9d0 con 0x7fb0c415e4e0 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 -- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c415e4e0 msgr2=0x7fb0c406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c415e4e0 0x7fb0c406c910 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80097f0 tx=0x7fb0b801c410 comp rx=0 tx=0).stop 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 -- 172.21.15.102:0/4066700630 shutdown_connections 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c406d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 --2- 172.21.15.102:0/4066700630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c415e4e0 0x7fb0c406c910 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 -- 172.21.15.102:0/4066700630 >> 172.21.15.102:0/4066700630 conn(0x7fb0c40d1f20 msgr2=0x7fb0c40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 -- 172.21.15.102:0/4066700630 shutdown_connections 2024-03-31T15:53:25.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.592+0000 7fb0cc800640 1 -- 172.21.15.102:0/4066700630 wait complete. 2024-03-31T15:53:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 Processor -- start 2024-03-31T15:53:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- start start 2024-03-31T15:53:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c41471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0c415e4e0 0x7fb0c41551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c416b1b0 con 0x7fb0c415e4e0 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c416b030 con 0x7fb0c406ce50 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c416b330 con 0x7fb0c40cdf20 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46986/0 (socket says 172.21.15.102:46986) 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c41471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 -- 172.21.15.102:0/3248406930 learned_addr learned my addr 172.21.15.102:0/3248406930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0ca800640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0c415e4e0 0x7fb0c41551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 -- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 msgr2=0x7fb0c41471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c41471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 -- 172.21.15.102:0/3248406930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0c415e4e0 msgr2=0x7fb0c41551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0c415e4e0 0x7fb0c41551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 -- 172.21.15.102:0/3248406930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c40cc340 con 0x7fb0c406ce50 2024-03-31T15:53:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0c9e00640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ae040 tx=0x7fb0b801ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b0050 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b80bc070 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b5870 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c415c950 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c40c96a0 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb0b80b5a10 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.596+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fb0b80b5d00 con 0x7fb0c406ce50 2024-03-31T15:53:25.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.600+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c40685f0 con 0x7fb0c406ce50 2024-03-31T15:53:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.600+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb0c40cc340 con 0x7fb0c406ce50 2024-03-31T15:53:25.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.668+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff0b59be-f0c8-44ea-82f1-5b644069371b", "id": 3} v 0) -- 0x7fb0c4147710 con 0x7fb0c406ce50 2024-03-31T15:53:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.672+0000 7fb0be000640 1 -- 172.21.15.102:0/3248406930 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff0b59be-f0c8-44ea-82f1-5b644069371b", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fb0b8016090 con 0x7fb0c406ce50 2024-03-31T15:53:25.666 INFO:teuthology.orchestra.run.smithi102.stdout:3 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.672+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 msgr2=0x7fb0c40cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.672+0000 7fb0cc800640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ae040 tx=0x7fb0b801ac50 comp rx=0 tx=0).stop 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 shutdown_connections 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0c415e4e0 0x7fb0c41551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0c40cdf20 0x7fb0c41471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 --2- 172.21.15.102:0/3248406930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c406ce50 0x7fb0c40cbe00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 >> 172.21.15.102:0/3248406930 conn(0x7fb0c40d1f20 msgr2=0x7fb0c40c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 shutdown_connections 2024-03-31T15:53:25.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.676+0000 7fb0cc800640 1 -- 172.21.15.102:0/3248406930 wait complete. 2024-03-31T15:53:25.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 5305284e-adc1-4c1f-882b-75769fe3ad3c 4 2024-03-31T15:53:25.700 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.708+0000 7f60dc64b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T15:53:25.761 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.768+0000 7f60dc64b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T15:53:25.772 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:25.780+0000 7f7df684c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 Processor -- start 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 -- start start 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f004815e4e0 0x7f004806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004815e8c0 con 0x7f004806ce50 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00480d0ff0 con 0x7f00480cdf20 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f004cc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f00480d12d0 con 0x7f004815e4e0 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0046800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f00480cdf20 0x7f00480ceeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51582/0 (socket says 172.21.15.102:51582) 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0046800640 1 -- 172.21.15.102:0/442930357 learned_addr learned my addr 172.21.15.102:0/442930357 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045e00640 1 --2- 172.21.15.102:0/442930357 >> v2:172.21.15.90:3300/0 conn(0x7f004815e4e0 0x7f004806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869714380 0 0) 0x7f004815e8c0 con 0x7f004806ce50 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f002c003530 con 0x7f004806ce50 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061366407 0 0) 0x7f00480d0ff0 con 0x7f00480cdf20 2024-03-31T15:53:25.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004815e8c0 con 0x7f00480cdf20 2024-03-31T15:53:25.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045e00640 1 -- 172.21.15.102:0/442930357 >> v1:172.21.15.102:6789/0 conn(0x7f00480cdf20 legacy=0x7f00480ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045e00640 1 -- 172.21.15.102:0/442930357 >> v1:172.21.15.90:6789/0 conn(0x7f004806ce50 legacy=0x7f00480ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.828+0000 7f0045e00640 1 -- 172.21.15.102:0/442930357 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00480cc710 con 0x7f004815e4e0 2024-03-31T15:53:25.825 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.828+0000 7f60dc64b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f0045e00640 1 --2- 172.21.15.102:0/442930357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f004815e4e0 0x7f004806c910 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00380097f0 tx=0x7f003801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318489a5a2c0306e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380b0070 con 0x7f004815e4e0 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00380ab600 con 0x7f004815e4e0 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f0045400640 1 -- 172.21.15.102:0/442930357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380ab9d0 con 0x7f004815e4e0 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 -- 172.21.15.102:0/442930357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f004815e4e0 msgr2=0x7f004806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 --2- 172.21.15.102:0/442930357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f004815e4e0 0x7f004806c910 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f00380097f0 tx=0x7f003801c410 comp rx=0 tx=0).stop 2024-03-31T15:53:25.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 -- 172.21.15.102:0/442930357 shutdown_connections 2024-03-31T15:53:25.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 --2- 172.21.15.102:0/442930357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f004815e4e0 0x7f004806c910 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 -- 172.21.15.102:0/442930357 >> 172.21.15.102:0/442930357 conn(0x7f00480d1f20 msgr2=0x7f00480d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 -- 172.21.15.102:0/442930357 shutdown_connections 2024-03-31T15:53:25.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.832+0000 7f004cc00640 1 -- 172.21.15.102:0/442930357 wait complete. 2024-03-31T15:53:25.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 Processor -- start 2024-03-31T15:53:25.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- start start 2024-03-31T15:53:25.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00480cdf20 0x7f0048155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f004815e4e0 0x7f0048156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f00480d0d90 con 0x7f00480cdf20 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f00480d0c10 con 0x7f004806ce50 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f004816c850 con 0x7f004815e4e0 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46996/0 (socket says 172.21.15.102:46996) 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 -- 172.21.15.102:0/1143474022 learned_addr learned my addr 172.21.15.102:0/1143474022 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0046800640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f004815e4e0 0x7f0048156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f003d400640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00480cdf20 0x7f0048155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 -- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f004815e4e0 msgr2=0x7f0048156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f004815e4e0 0x7f0048156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 -- 172.21.15.102:0/1143474022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00480cdf20 msgr2=0x7f0048155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00480cdf20 0x7f0048155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 -- 172.21.15.102:0/1143474022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004814f120 con 0x7f004806ce50 2024-03-31T15:53:25.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f0045e00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00380ae040 tx=0x7f003801ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380b0050 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00380bc070 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038006e20 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00480c9d20 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.836+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00480ca2e0 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.840+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f00380b5870 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.840+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f00380b7960 con 0x7f004806ce50 2024-03-31T15:53:25.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.840+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00480685f0 con 0x7f004806ce50 2024-03-31T15:53:25.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.840+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f004814f120 con 0x7f004806ce50 2024-03-31T15:53:25.838 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:25.844+0000 7f7df684c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-31T15:53:25.899 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:25.908+0000 7f7df684c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-31T15:53:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.908+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5305284e-adc1-4c1f-882b-75769fe3ad3c", "id": 4} v 0) -- 0x7f00480b2bf0 con 0x7f004806ce50 2024-03-31T15:53:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f003ea00640 1 -- 172.21.15.102:0/1143474022 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5305284e-adc1-4c1f-882b-75769fe3ad3c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f0038016090 con 0x7f004806ce50 2024-03-31T15:53:25.906 INFO:teuthology.orchestra.run.smithi102.stdout:4 2024-03-31T15:53:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 msgr2=0x7f004814ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00380ae040 tx=0x7f003801ac50 comp rx=0 tx=0).stop 2024-03-31T15:53:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 shutdown_connections 2024-03-31T15:53:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f004815e4e0 0x7f0048156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00480cdf20 0x7f0048155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 --2- 172.21.15.102:0/1143474022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f004806ce50 0x7f004814ebe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.912+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 >> 172.21.15.102:0/1143474022 conn(0x7f00480d1f20 msgr2=0x7f00480d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.916+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 shutdown_connections 2024-03-31T15:53:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:25.916+0000 7f004cc00640 1 -- 172.21.15.102:0/1143474022 wait complete. 2024-03-31T15:53:25.908 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.916+0000 7f60dc64b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T15:53:25.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 31d64299-ceb7-4ab7-b692-45a8b7bf01df 5 2024-03-31T15:53:25.968 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:25.976+0000 7f60dc64b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T15:53:25.993 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.000+0000 7f7df684c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-31T15:53:26.055 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:26.064+0000 7f60dc64b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T15:53:26.055 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.064+0000 7f7df684c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-31T15:53:26.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 Processor -- start 2024-03-31T15:53:26.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 -- start start 2024-03-31T15:53:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24e80ccb30 0x7f24e806e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f24e8162400 0x7f24e8162850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e8162d90 con 0x7f24e806e970 2024-03-31T15:53:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f24e8072de0 con 0x7f24e8162400 2024-03-31T15:53:26.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ef600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24e80730c0 con 0x7f24e80ccb30 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24e80ccb30 0x7f24e806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24e80ccb30 0x7f24e806e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47004/0 (socket says 172.21.15.102:47004) 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 -- 172.21.15.102:0/785591302 learned_addr learned my addr 172.21.15.102:0/785591302 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ed600640 1 --2- 172.21.15.102:0/785591302 >> v2:172.21.15.90:3300/0 conn(0x7f24e8162400 0x7f24e8162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 -- 172.21.15.102:0/785591302 >> v2:172.21.15.90:3300/0 conn(0x7f24e8162400 msgr2=0x7f24e8162850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 --2- 172.21.15.102:0/785591302 >> v2:172.21.15.90:3300/0 conn(0x7f24e8162400 0x7f24e8162850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 -- 172.21.15.102:0/785591302 >> v1:172.21.15.90:6789/0 conn(0x7f24e806e970 legacy=0x7f24e806b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.068+0000 7f24ecc00640 1 -- 172.21.15.102:0/785591302 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e80733a0 con 0x7f24e80ccb30 2024-03-31T15:53:26.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ecc00640 1 --2- 172.21.15.102:0/785591302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e80ccb30 0x7f24e806e430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f24e80ce930 tx=0x7f24dc01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1573dfbe7f2a1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24e7400640 1 -- 172.21.15.102:0/785591302 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0b1070 con 0x7f24e80ccb30 2024-03-31T15:53:26.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24e7400640 1 -- 172.21.15.102:0/785591302 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc0ac640 con 0x7f24e80ccb30 2024-03-31T15:53:26.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24e7400640 1 -- 172.21.15.102:0/785591302 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0aca10 con 0x7f24e80ccb30 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 -- 172.21.15.102:0/785591302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e80ccb30 msgr2=0x7f24e806e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 --2- 172.21.15.102:0/785591302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e80ccb30 0x7f24e806e430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f24e80ce930 tx=0x7f24dc01c490 comp rx=0 tx=0).stop 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 -- 172.21.15.102:0/785591302 shutdown_connections 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 --2- 172.21.15.102:0/785591302 >> v2:172.21.15.90:3300/0 conn(0x7f24e8162400 0x7f24e8162850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 --2- 172.21.15.102:0/785591302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e80ccb30 0x7f24e806e430 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 -- 172.21.15.102:0/785591302 >> 172.21.15.102:0/785591302 conn(0x7f24e80b29a0 msgr2=0x7f24e80b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 -- 172.21.15.102:0/785591302 shutdown_connections 2024-03-31T15:53:26.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.072+0000 7f24ef600640 1 -- 172.21.15.102:0/785591302 wait complete. 2024-03-31T15:53:26.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 Processor -- start 2024-03-31T15:53:26.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- start start 2024-03-31T15:53:26.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e80ccb30 0x7f24e80c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8162400 0x7f24e80c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8072ba0 con 0x7f24e8162400 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24e8072a20 con 0x7f24e806e970 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f24e80d0900 con 0x7f24e80ccb30 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47012/0 (socket says 172.21.15.102:47012) 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 -- 172.21.15.102:0/2803051207 learned_addr learned my addr 172.21.15.102:0/2803051207 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e7e00640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e80ccb30 0x7f24e80c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ed600640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8162400 0x7f24e80c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 -- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e80ccb30 msgr2=0x7f24e80c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e80ccb30 0x7f24e80c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 -- 172.21.15.102:0/2803051207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8162400 msgr2=0x7f24e80c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8162400 0x7f24e80c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 -- 172.21.15.102:0/2803051207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e80c8910 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ecc00640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0af040 tx=0x7f24dc019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0b1040 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc0b5420 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0b57f0 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e815b190 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e815b750 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24dc0b5990 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f24dc0b5c80 con 0x7f24e806e970 2024-03-31T15:53:26.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.076+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e8068d00 con 0x7f24e806e970 2024-03-31T15:53:26.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.080+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f24dc016110 con 0x7f24e806e970 2024-03-31T15:53:26.118 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:26.124+0000 7f60dc64b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T15:53:26.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.148+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "31d64299-ceb7-4ab7-b692-45a8b7bf01df", "id": 5} v 0) -- 0x7f24e80c79c0 con 0x7f24e806e970 2024-03-31T15:53:26.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.152+0000 7f24e4c00640 1 -- 172.21.15.102:0/2803051207 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "31d64299-ceb7-4ab7-b692-45a8b7bf01df", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f24dc0be400 con 0x7f24e806e970 2024-03-31T15:53:26.147 INFO:teuthology.orchestra.run.smithi102.stdout:5 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 msgr2=0x7f24e814fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0af040 tx=0x7f24dc019230 comp rx=0 tx=0).stop 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 shutdown_connections 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24e8162400 0x7f24e80c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24e80ccb30 0x7f24e80c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 --2- 172.21.15.102:0/2803051207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e806e970 0x7f24e814fde0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 >> 172.21.15.102:0/2803051207 conn(0x7f24e80b29a0 msgr2=0x7f24e80b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 shutdown_connections 2024-03-31T15:53:26.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f24ef600640 1 -- 172.21.15.102:0/2803051207 wait complete. 2024-03-31T15:53:26.149 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.156+0000 7f7df684c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-31T15:53:26.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 4ee5275f-1b33-4e41-8337-316a3de87dfd 6 2024-03-31T15:53:26.218 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.224+0000 7f7df684c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-31T15:53:26.283 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:26.292+0000 7f60dc64b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T15:53:26.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.308+0000 7fd592c00640 1 Processor -- start 2024-03-31T15:53:26.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.308+0000 7fd592c00640 1 -- start start 2024-03-31T15:53:26.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd592c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 0x7fd58c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd592c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58c0cefc0 con 0x7fd58c0cd640 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd592c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58c0cf2a0 con 0x7fd58c0d2b20 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd592c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c0aed70 con 0x7fd58c15c320 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd58c0d2b20 0x7fd58c0ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51596/0 (socket says 172.21.15.102:51596) 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58b400640 1 -- 172.21.15.102:0/3807381298 learned_addr learned my addr 172.21.15.102:0/3807381298 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-31T15:53:26.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58be00640 1 --2- 172.21.15.102:0/3807381298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 0x7fd58c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517846705 0 0) 0x7fd58c0cf2a0 con 0x7fd58c0d2b20 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd570003530 con 0x7fd58c0d2b20 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471908908 0 0) 0x7fd58c0cefc0 con 0x7fd58c0cd640 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd58c0cf2a0 con 0x7fd58c0cd640 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58be00640 1 -- 172.21.15.102:0/3807381298 >> v1:172.21.15.102:6789/0 conn(0x7fd58c0d2b20 legacy=0x7fd58c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58be00640 1 -- 172.21.15.102:0/3807381298 >> v1:172.21.15.90:6789/0 conn(0x7fd58c0cd640 legacy=0x7fd58c0af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.312+0000 7fd58be00640 1 -- 172.21.15.102:0/3807381298 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c0cf580 con 0x7fd58c15c320 2024-03-31T15:53:26.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd58be00640 1 --2- 172.21.15.102:0/3807381298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 0x7fd58c0cc9f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0097f0 tx=0x7fd57c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd0fad5dac65ed2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0b2070 con 0x7fd58c15c320 2024-03-31T15:53:26.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c0a9070 con 0x7fd58c15c320 2024-03-31T15:53:26.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd58aa00640 1 -- 172.21.15.102:0/3807381298 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0ad880 con 0x7fd58c15c320 2024-03-31T15:53:26.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- 172.21.15.102:0/3807381298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 msgr2=0x7fd58c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 --2- 172.21.15.102:0/3807381298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 0x7fd58c0cc9f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0097f0 tx=0x7fd57c01c450 comp rx=0 tx=0).stop 2024-03-31T15:53:26.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- 172.21.15.102:0/3807381298 shutdown_connections 2024-03-31T15:53:26.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 --2- 172.21.15.102:0/3807381298 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c15c320 0x7fd58c0cc9f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- 172.21.15.102:0/3807381298 >> 172.21.15.102:0/3807381298 conn(0x7fd58c0d1f20 msgr2=0x7fd58c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- 172.21.15.102:0/3807381298 shutdown_connections 2024-03-31T15:53:26.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- 172.21.15.102:0/3807381298 wait complete. 2024-03-31T15:53:26.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 Processor -- start 2024-03-31T15:53:26.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.316+0000 7fd592c00640 1 -- start start 2024-03-31T15:53:26.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c0cd640 0x7fd58c14cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 0x7fd58c14edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c169ed0 con 0x7fd58c0cd640 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c0b03f0 con 0x7fd58c15c320 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c16a050 con 0x7fd58c0d2b20 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd58b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd58b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37134/0 (socket says 172.21.15.102:37134) 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd58b400640 1 -- 172.21.15.102:0/1752688779 learned_addr learned my addr 172.21.15.102:0/1752688779 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 0x7fd58c14edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd58be00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c0cd640 0x7fd58c14cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 -- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 msgr2=0x7fd58c06d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 -- 172.21.15.102:0/1752688779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c0cd640 msgr2=0x7fd58c14cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c0cd640 0x7fd58c14cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 -- 172.21.15.102:0/1752688779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c15eaf0 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd58b400640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd590c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 0x7fd58c14edd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0cdad0 tx=0x7fd58400cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584016020 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd584006d60 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584003b80 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c15edd0 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c15f390 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd584005460 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fd584005750 con 0x7fd58c15c320 2024-03-31T15:53:26.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.320+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd58c0685f0 con 0x7fd58c15c320 2024-03-31T15:53:26.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.324+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd584020020 con 0x7fd58c15c320 2024-03-31T15:53:26.354 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:26.360+0000 7f60dc64b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T15:53:26.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.388+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4ee5275f-1b33-4e41-8337-316a3de87dfd", "id": 6} v 0) -- 0x7fd58c15f560 con 0x7fd58c15c320 2024-03-31T15:53:26.389 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.396+0000 7f7df684c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-31T15:53:26.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.396+0000 7fd583e00640 1 -- 172.21.15.102:0/1752688779 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4ee5275f-1b33-4e41-8337-316a3de87dfd", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fd584003d20 con 0x7fd58c15c320 2024-03-31T15:53:26.390 INFO:teuthology.orchestra.run.smithi102.stdout:6 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 msgr2=0x7fd58c14edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 0x7fd58c14edd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0cdad0 tx=0x7fd58400cb90 comp rx=0 tx=0).stop 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 shutdown_connections 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd58c15c320 0x7fd58c14edd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd58c0d2b20 0x7fd58c06d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 --2- 172.21.15.102:0/1752688779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c0cd640 0x7fd58c14cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 >> 172.21.15.102:0/1752688779 conn(0x7fd58c0d1f20 msgr2=0x7fd58c0cfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 shutdown_connections 2024-03-31T15:53:26.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.400+0000 7fd592c00640 1 -- 172.21.15.102:0/1752688779 wait complete. 2024-03-31T15:53:26.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd new 636531f9-c5c4-4648-92b5-f3747e70afa4 7 2024-03-31T15:53:26.464 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:26.472+0000 7f7df684c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-31T15:53:26.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 Processor -- start 2024-03-31T15:53:26.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 -- start start 2024-03-31T15:53:26.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 0x7fc5a406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a40cbc10 con 0x7fc5a40cd450 2024-03-31T15:53:26.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a406f5b0 con 0x7fc5a4072b40 2024-03-31T15:53:26.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5aac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a406f890 con 0x7fc5a415ae90 2024-03-31T15:53:26.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 0x7fc5a406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a8c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc5a4072b40 0x7fc5a40cc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51604/0 (socket says 172.21.15.102:51604) 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a8c00640 1 -- 172.21.15.102:0/3726712084 learned_addr learned my addr 172.21.15.102:0/3726712084 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a2a00640 1 -- 172.21.15.102:0/3726712084 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2778328422 0 0) 0x7fc5a406f5b0 con 0x7fc5a4072b40 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a2a00640 1 -- 172.21.15.102:0/3726712084 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc588003530 con 0x7fc5a4072b40 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a3e00640 1 -- 172.21.15.102:0/3726712084 >> v1:172.21.15.102:6789/0 conn(0x7fc5a4072b40 legacy=0x7fc5a40cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a3e00640 1 -- 172.21.15.102:0/3726712084 >> v1:172.21.15.90:6789/0 conn(0x7fc5a40cd450 legacy=0x7fc5a4072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.552+0000 7fc5a3e00640 1 -- 172.21.15.102:0/3726712084 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a40cbeb0 con 0x7fc5a415ae90 2024-03-31T15:53:26.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5a3e00640 1 --2- 172.21.15.102:0/3726712084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 0x7fc5a406ec40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097f0 tx=0x7fc59401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bc2a1aeff01a47d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5a2a00640 1 -- 172.21.15.102:0/3726712084 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940b5070 con 0x7fc5a415ae90 2024-03-31T15:53:26.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5a2a00640 1 -- 172.21.15.102:0/3726712084 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5940a89d0 con 0x7fc5a415ae90 2024-03-31T15:53:26.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5a2a00640 1 -- 172.21.15.102:0/3726712084 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940a8da0 con 0x7fc5a415ae90 2024-03-31T15:53:26.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 -- 172.21.15.102:0/3726712084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 msgr2=0x7fc5a406ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 --2- 172.21.15.102:0/3726712084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 0x7fc5a406ec40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097f0 tx=0x7fc59401ecd0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 -- 172.21.15.102:0/3726712084 shutdown_connections 2024-03-31T15:53:26.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 --2- 172.21.15.102:0/3726712084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a415ae90 0x7fc5a406ec40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 -- 172.21.15.102:0/3726712084 >> 172.21.15.102:0/3726712084 conn(0x7fc5a40b29a0 msgr2=0x7fc5a40b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 -- 172.21.15.102:0/3726712084 shutdown_connections 2024-03-31T15:53:26.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.556+0000 7fc5aac00640 1 -- 172.21.15.102:0/3726712084 wait complete. 2024-03-31T15:53:26.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 Processor -- start 2024-03-31T15:53:26.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- start start 2024-03-31T15:53:26.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a40cd450 0x7fc5a4152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5a415ae90 0x7fc5a415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a415b3f0 con 0x7fc5a40cd450 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a415b270 con 0x7fc5a415ae90 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a406f180 con 0x7fc5a4072b40 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5a415ae90 0x7fc5a415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.102:37138/0 (socket says 172.21.15.102:37138) 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 -- 172.21.15.102:0/1486333069 learned_addr learned my addr 172.21.15.102:0/1486333069 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3400640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a40cd450 0x7fc5a4152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 -- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5a415ae90 msgr2=0x7fc5a415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5a415ae90 0x7fc5a415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 -- 172.21.15.102:0/1486333069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a40cd450 msgr2=0x7fc5a4152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a40cd450 0x7fc5a4152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 -- 172.21.15.102:0/1486333069 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a40b4f10 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5a3e00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097f0 tx=0x7fc59401ba70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940b5040 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5940bf070 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a40b51f0 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940b0960 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a40b57b0 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc5940b0b00 con 0x7fc5a4072b40 2024-03-31T15:53:26.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fc5940c4b90 con 0x7fc5a4072b40 2024-03-31T15:53:26.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.560+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a4068d00 con 0x7fc5a4072b40 2024-03-31T15:53:26.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.564+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc594018830 con 0x7fc5a4072b40 2024-03-31T15:53:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.632+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd new", "uuid": "636531f9-c5c4-4648-92b5-f3747e70afa4", "id": 7} v 0) -- 0x7fc5a415a400 con 0x7fc5a4072b40 2024-03-31T15:53:26.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.636+0000 7fc59be00640 1 -- 172.21.15.102:0/1486333069 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "636531f9-c5c4-4648-92b5-f3747e70afa4", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fc5940c4460 con 0x7fc5a4072b40 2024-03-31T15:53:26.631 INFO:teuthology.orchestra.run.smithi102.stdout:7 2024-03-31T15:53:26.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 msgr2=0x7fc5a40b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc5940097f0 tx=0x7fc59401ba70 comp rx=0 tx=0).stop 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 shutdown_connections 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5a415ae90 0x7fc5a415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a40cd450 0x7fc5a4152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 --2- 172.21.15.102:0/1486333069 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc5a4072b40 0x7fc5a40b48f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 >> 172.21.15.102:0/1486333069 conn(0x7fc5a40b29a0 msgr2=0x7fc5a40b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 shutdown_connections 2024-03-31T15:53:26.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:53:26.640+0000 7fc5aac00640 1 -- 172.21.15.102:0/1486333069 wait complete. 2024-03-31T15:53:26.649 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-31T15:53:26.649 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-31T15:53:26.652 INFO:tasks.ceph.osd.0:Started 2024-03-31T15:53:26.652 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-31T15:53:26.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-31T15:53:26.657 INFO:tasks.ceph.osd.1:Started 2024-03-31T15:53:26.657 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-31T15:53:26.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-31T15:53:26.660 INFO:tasks.ceph.osd.2:Started 2024-03-31T15:53:26.660 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-31T15:53:26.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-31T15:53:26.663 INFO:tasks.ceph.osd.3:Started 2024-03-31T15:53:26.664 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-31T15:53:26.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-31T15:53:26.667 INFO:tasks.ceph.osd.4:Started 2024-03-31T15:53:26.668 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-31T15:53:26.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-31T15:53:26.671 INFO:tasks.ceph.osd.5:Started 2024-03-31T15:53:26.671 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-31T15:53:26.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-31T15:53:26.674 INFO:tasks.ceph.osd.6:Started 2024-03-31T15:53:26.674 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-31T15:53:26.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-31T15:53:26.678 INFO:tasks.ceph.osd.7:Started 2024-03-31T15:53:26.678 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:26.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 Processor -- start 2024-03-31T15:53:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 -- start start 2024-03-31T15:53:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 0x7ff11015fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1100cca30 0x7ff1100ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff110160350 con 0x7ff1100d2680 2024-03-31T15:53:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff110160bf0 con 0x7ff1100cca30 2024-03-31T15:53:26.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff115000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff110160630 con 0x7ff11015c120 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff10e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1100cca30 0x7ff1100ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff10e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff1100cca30 0x7ff1100ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41258/0 (socket says 172.21.15.90:41258) 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff10e800640 1 -- 172.21.15.90:0/2237953374 learned_addr learned my addr 172.21.15.90:0/2237953374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.880+0000 7ff10de00640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 0x7ff11015fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10e800640 1 -- 172.21.15.90:0/2237953374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 msgr2=0x7ff11015fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10e800640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 0x7ff11015fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10e800640 1 -- 172.21.15.90:0/2237953374 >> v1:172.21.15.102:6789/0 conn(0x7ff1100d2680 legacy=0x7ff1100b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10e800640 1 -- 172.21.15.90:0/2237953374 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff110160910 con 0x7ff1100cca30 2024-03-31T15:53:26.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10de00640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 0x7ff11015fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10e800640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100cca30 0x7ff1100ccea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff108009e50 tx=0x7ff10801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cb3971e124fc78 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10d400640 1 -- 172.21.15.90:0/2237953374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1080a5770 con 0x7ff1100cca30 2024-03-31T15:53:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10d400640 1 -- 172.21.15.90:0/2237953374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1080a5910 con 0x7ff1100cca30 2024-03-31T15:53:26.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff10d400640 1 -- 172.21.15.90:0/2237953374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1080a5ce0 con 0x7ff1100cca30 2024-03-31T15:53:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 -- 172.21.15.90:0/2237953374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100cca30 msgr2=0x7ff1100ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100cca30 0x7ff1100ccea0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff108009e50 tx=0x7ff10801c9a0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 -- 172.21.15.90:0/2237953374 shutdown_connections 2024-03-31T15:53:26.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100cca30 0x7ff1100ccea0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 --2- 172.21.15.90:0/2237953374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff11015c120 0x7ff11015fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 -- 172.21.15.90:0/2237953374 >> 172.21.15.90:0/2237953374 conn(0x7ff11009aa40 msgr2=0x7ff11009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 -- 172.21.15.90:0/2237953374 shutdown_connections 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.884+0000 7ff115000640 1 -- 172.21.15.90:0/2237953374 wait complete. 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 Processor -- start 2024-03-31T15:53:26.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 -- start start 2024-03-31T15:53:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1100cca30 0x7ff11014b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff11015c120 0x7ff110154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100b2f70 con 0x7ff1100d2680 2024-03-31T15:53:26.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1100b2df0 con 0x7ff11015c120 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff110169040 con 0x7ff1100cca30 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41260/0 (socket says 172.21.15.90:41260) 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 -- 172.21.15.90:0/1947924777 learned_addr learned my addr 172.21.15.90:0/1947924777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff10de00640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1100cca30 0x7ff11014b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff10e800640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff11015c120 0x7ff110154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 -- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1100cca30 msgr2=0x7ff11014b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1100cca30 0x7ff11014b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 -- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff11015c120 msgr2=0x7ff110154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff11015c120 0x7ff110154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 -- 172.21.15.90:0/1947924777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11014b5c0 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff107e00640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c3a0 tx=0x7ff0f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80052d0 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8005470 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1101f5600 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.888+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8007260 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.892+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1101f5bc0 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.892+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff0f8007400 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.892+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff0f80039c0 con 0x7ff1100d2680 2024-03-31T15:53:26.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.892+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1100686d0 con 0x7ff1100d2680 2024-03-31T15:53:26.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.892+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff0f8003ce0 con 0x7ff1100d2680 2024-03-31T15:53:26.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.948+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1100b1b60 con 0x7ff1100d2680 2024-03-31T15:53:26.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.948+0000 7ff106000640 1 -- 172.21.15.90:0/1947924777 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff0f80067e0 con 0x7ff1100d2680 2024-03-31T15:53:26.942 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:26.942 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:26.636888+0000","last_up_change":"0.000000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 msgr2=0x7ff11015acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c3a0 tx=0x7ff0f800c860 comp rx=0 tx=0).stop 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 shutdown_connections 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff11015c120 0x7ff110154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff1100d2680 0x7ff11015acb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 --2- 172.21.15.90:0/1947924777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1100cca30 0x7ff11014b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 >> 172.21.15.90:0/1947924777 conn(0x7ff11009aa40 msgr2=0x7ff11006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 shutdown_connections 2024-03-31T15:53:26.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:26.952+0000 7ff115000640 1 -- 172.21.15.90:0/1947924777 wait complete. 2024-03-31T15:53:26.961 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-31T15:53:26.961 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-31T15:53:27.091 INFO:tasks.ceph.mgr.y.smithi090.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-31T15:53:27.091 INFO:tasks.ceph.mgr.y.smithi090.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-31T15:53:27.091 INFO:tasks.ceph.mgr.y.smithi090.stderr: from numpy import show_config as show_numpy_config 2024-03-31T15:53:27.100 INFO:tasks.ceph.mgr.x.smithi102.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-31T15:53:27.100 INFO:tasks.ceph.mgr.x.smithi102.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-31T15:53:27.100 INFO:tasks.ceph.mgr.x.smithi102.stderr: from numpy import show_config as show_numpy_config 2024-03-31T15:53:27.146 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:27.156+0000 7f60dc64b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T15:53:27.159 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:27.168+0000 7f7df684c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-31T15:53:27.295 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:27.304+0000 7f60dc64b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:53:27.308 INFO:tasks.ceph.osd.4.smithi102.stderr:2024-03-31T15:53:27.316+0000 7f644296e7c0 -1 Falling back to public interface 2024-03-31T15:53:27.308 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:27.316+0000 7f7df684c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:53:27.311 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-31T15:53:27.320+0000 7f7c29a2b7c0 -1 Falling back to public interface 2024-03-31T15:53:27.323 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-31T15:53:27.332+0000 7f806342c7c0 -1 Falling back to public interface 2024-03-31T15:53:27.324 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-31T15:53:27.332+0000 7f6721d6e7c0 -1 Falling back to public interface 2024-03-31T15:53:27.324 INFO:tasks.ceph.osd.5.smithi102.stderr:2024-03-31T15:53:27.332+0000 7fe9a69ab7c0 -1 Falling back to public interface 2024-03-31T15:53:27.324 INFO:tasks.ceph.osd.7.smithi102.stderr:2024-03-31T15:53:27.332+0000 7f4ef062b7c0 -1 Falling back to public interface 2024-03-31T15:53:27.339 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-31T15:53:27.348+0000 7f2838a1b7c0 -1 Falling back to public interface 2024-03-31T15:53:27.360 INFO:tasks.ceph.osd.6.smithi102.stderr:2024-03-31T15:53:27.368+0000 7f492ee5d7c0 -1 Falling back to public interface 2024-03-31T15:53:27.501 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:27.508+0000 7f7df684c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T15:53:27.505 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:27.512+0000 7f60dc64b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-31T15:53:27.574 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:27.580+0000 7f7df684c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T15:53:27.610 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:27.620+0000 7f60dc64b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-31T15:53:27.726 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:27.732+0000 7f7df684c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:53:27.760 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:27.768+0000 7f60dc64b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-31T15:53:27.766 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:27.894 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.900+0000 7fe549600640 1 Processor -- start 2024-03-31T15:53:27.895 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 -- start start 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe54413f280 0x7fe5441419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe544141f10 0x7fe5440ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54413f660 con 0x7fe5440cedf0 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe5440cfba0 con 0x7fe544141f10 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe549600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe5440cf5e0 con 0x7fe54413f280 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe53a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe544141f10 0x7fe5440ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe543400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe5440cedf0 0x7fe54415fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:59846/0 (socket says 172.21.15.90:59846) 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe543400640 1 -- 172.21.15.90:0/1205825713 learned_addr learned my addr 172.21.15.90:0/1205825713 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:27.896 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe542a00640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe54413f280 0x7fe5441419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:27.897 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe53a000640 1 -- 172.21.15.90:0/1205825713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe54413f280 msgr2=0x7fe5441419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:27.897 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe53a000640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe54413f280 0x7fe5441419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.897 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe53a000640 1 -- 172.21.15.90:0/1205825713 >> v1:172.21.15.90:6789/0 conn(0x7fe5440cedf0 legacy=0x7fe54415fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:27.897 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe53a000640 1 -- 172.21.15.90:0/1205825713 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5440cf8c0 con 0x7fe544141f10 2024-03-31T15:53:27.897 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.904+0000 7fe542000640 1 -- 172.21.15.90:0/1205825713 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268880375 0 0) 0x7fe53c001120 con 0x7fe5440cedf0 2024-03-31T15:53:27.899 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe53a000640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe5440ce8b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0097c0 tx=0x7fe52c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0be2355630ec04b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:27.899 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe542000640 1 -- 172.21.15.90:0/1205825713 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c0b2070 con 0x7fe544141f10 2024-03-31T15:53:27.899 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe542000640 1 -- 172.21.15.90:0/1205825713 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52c0ad640 con 0x7fe544141f10 2024-03-31T15:53:27.899 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe542000640 1 -- 172.21.15.90:0/1205825713 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c0ada10 con 0x7fe544141f10 2024-03-31T15:53:27.900 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- 172.21.15.90:0/1205825713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 msgr2=0x7fe5440ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:27.900 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe5440ce8b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0097c0 tx=0x7fe52c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.900 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- 172.21.15.90:0/1205825713 shutdown_connections 2024-03-31T15:53:27.900 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe5440ce8b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.901 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 --2- 172.21.15.90:0/1205825713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe54413f280 0x7fe5441419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.901 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- 172.21.15.90:0/1205825713 >> 172.21.15.90:0/1205825713 conn(0x7fe54406b620 msgr2=0x7fe54406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:27.901 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- 172.21.15.90:0/1205825713 shutdown_connections 2024-03-31T15:53:27.901 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- 172.21.15.90:0/1205825713 wait complete. 2024-03-31T15:53:27.902 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 Processor -- start 2024-03-31T15:53:27.902 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.908+0000 7fe549600640 1 -- start start 2024-03-31T15:53:27.903 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:27.903 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 0x7fe54413ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:27.903 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe54415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe544169010 con 0x7fe544141f10 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe544168e90 con 0x7fe54413f280 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe544169190 con 0x7fe5440cedf0 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe543400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe54415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:51030/0 (socket says 172.21.15.90:51030) 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 0x7fe54413ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:27.904 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 -- 172.21.15.90:0/2547342027 learned_addr learned my addr 172.21.15.90:0/2547342027 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 -- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 msgr2=0x7fe54413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 0x7fe54413ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 -- 172.21.15.90:0/2547342027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 msgr2=0x7fe54415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe54415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 -- 172.21.15.90:0/2547342027 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54413b1c0 con 0x7fe5440cedf0 2024-03-31T15:53:27.906 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53a000640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 0x7fe54413ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe543400640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe54415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe542a00640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe53400efc0 tx=0x7fe53400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534018020 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe534014480 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534014850 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54413d380 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54413d940 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe53401b020 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe534014d80 con 0x7fe5440cedf0 2024-03-31T15:53:27.907 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.912+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe544068f20 con 0x7fe5440cedf0 2024-03-31T15:53:27.908 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.916+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe534022070 con 0x7fe5440cedf0 2024-03-31T15:53:27.964 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5440b4be0 con 0x7fe5440cedf0 2024-03-31T15:53:27.964 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe53b400640 1 -- 172.21.15.90:0/2547342027 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe534010040 con 0x7fe5440cedf0 2024-03-31T15:53:27.965 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-31T15:53:27.965 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":10,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:26.636888+0000","last_up_change":"0.000000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 msgr2=0x7fe54415b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe53400efc0 tx=0x7fe53400c490 comp rx=0 tx=0).stop 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 shutdown_connections 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe544141f10 0x7fe54415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe54413f280 0x7fe54413ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 --2- 172.21.15.90:0/2547342027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5440cedf0 0x7fe54415b660 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 >> 172.21.15.90:0/2547342027 conn(0x7fe54406b620 msgr2=0x7fe544166690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 shutdown_connections 2024-03-31T15:53:27.966 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:27.972+0000 7fe549600640 1 -- 172.21.15.90:0/2547342027 wait complete. 2024-03-31T15:53:27.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-31T15:53:28.414 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:28.420+0000 7f7df684c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T15:53:28.475 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:28.484+0000 7f60dc64b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-31T15:53:28.499 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:28.508+0000 7f7df684c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T15:53:28.561 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:28.568+0000 7f60dc64b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-31T15:53:28.702 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:28.708+0000 7f7df684c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T15:53:28.763 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:28.772+0000 7f60dc64b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-31T15:53:28.785 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:28.792+0000 7f7df684c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T15:53:28.847 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:28.856+0000 7f60dc64b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-31T15:53:29.113 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:29.120+0000 7f7df684c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T15:53:29.174 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:29.184+0000 7f60dc64b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-31T15:53:29.184 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:29.192+0000 7f7df684c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T15:53:29.248 INFO:tasks.ceph.mgr.y.smithi090.stderr:2024-03-31T15:53:29.256+0000 7f60dc64b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-31T15:53:30.206 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:30.212+0000 7f7d9be00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-31T15:53:30.207 INFO:tasks.ceph.mgr.x.smithi102.stderr:2024-03-31T15:53:30.212+0000 7f7d9be00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-31T15:53:31.097 INFO:tasks.ceph.osd.7.smithi102.stderr:2024-03-31T15:53:31.104+0000 7f4ef062b7c0 -1 osd.7 0 log_to_monitors true 2024-03-31T15:53:31.134 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-31T15:53:31.140+0000 7f7c29a2b7c0 -1 osd.2 0 log_to_monitors true 2024-03-31T15:53:31.227 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-31T15:53:31.236+0000 7f6721d6e7c0 -1 osd.3 0 log_to_monitors true 2024-03-31T15:53:31.236 INFO:tasks.ceph.osd.4.smithi102.stderr:2024-03-31T15:53:31.244+0000 7f644296e7c0 -1 osd.4 0 log_to_monitors true 2024-03-31T15:53:31.244 INFO:tasks.ceph.osd.5.smithi102.stderr:2024-03-31T15:53:31.252+0000 7fe9a69ab7c0 -1 osd.5 0 log_to_monitors true 2024-03-31T15:53:31.284 INFO:tasks.ceph.osd.6.smithi102.stderr:2024-03-31T15:53:31.292+0000 7f492ee5d7c0 -1 osd.6 0 log_to_monitors true 2024-03-31T15:53:31.500 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-31T15:53:31.508+0000 7f2838a1b7c0 -1 osd.1 0 log_to_monitors true 2024-03-31T15:53:31.508 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-31T15:53:31.516+0000 7f806342c7c0 -1 osd.0 0 log_to_monitors true 2024-03-31T15:53:33.174 INFO:tasks.ceph.osd.7.smithi102.stderr:2024-03-31T15:53:33.180+0000 7f4eeaa00640 -1 osd.7 0 waiting for initial osdmap 2024-03-31T15:53:33.200 INFO:tasks.ceph.osd.7.smithi102.stderr:2024-03-31T15:53:33.208+0000 7f4ee0a00640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:33.715 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-31T15:53:33.724+0000 7f671d600640 -1 osd.3 0 waiting for initial osdmap 2024-03-31T15:53:33.751 INFO:tasks.ceph.osd.3.smithi090.stderr:2024-03-31T15:53:33.760+0000 7f6712c00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:33.786 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-31T15:53:33.792+0000 7f7c24000640 -1 osd.2 0 waiting for initial osdmap 2024-03-31T15:53:33.836 INFO:tasks.ceph.osd.2.smithi090.stderr:2024-03-31T15:53:33.844+0000 7f7c1a000640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:34.642 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-31T15:53:34.648+0000 7f805e400640 -1 osd.0 0 waiting for initial osdmap 2024-03-31T15:53:34.662 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-31T15:53:34.668+0000 7f2834200640 -1 osd.1 0 waiting for initial osdmap 2024-03-31T15:53:34.668 INFO:tasks.ceph.osd.0.smithi090.stderr:2024-03-31T15:53:34.676+0000 7f8053a00640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:34.687 INFO:tasks.ceph.osd.1.smithi090.stderr:2024-03-31T15:53:34.696+0000 7f2829800640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:34.787 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:34.912 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 Processor -- start 2024-03-31T15:53:34.912 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 -- start start 2024-03-31T15:53:34.919 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f815c159a70 0x7f815c0cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.919 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f815c0cf470 con 0x7f815c159e50 2024-03-31T15:53:34.919 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f815c0b3190 con 0x7f815c1603b0 2024-03-31T15:53:34.919 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8161800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f815c0cfc50 con 0x7f815c159a70 2024-03-31T15:53:34.919 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8153e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f815c159e50 0x7f815c15fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51012/0 (socket says 172.21.15.90:51012) 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8153e00640 1 -- 172.21.15.90:0/159989255 learned_addr learned my addr 172.21.15.90:0/159989255 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815aa00640 1 --2- 172.21.15.90:0/159989255 >> v2:172.21.15.102:3300/0 conn(0x7f815c159a70 0x7f815c0cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650957147 0 0) 0x7f815c0cf470 con 0x7f815c159e50 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8140003530 con 0x7f815c159e50 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2140370105 0 0) 0x7f8140003530 con 0x7f815c159e50 2024-03-31T15:53:34.920 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 >> v2:172.21.15.102:3300/0 conn(0x7f815c159a70 msgr2=0x7f815c0cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 --2- 172.21.15.90:0/159989255 >> v2:172.21.15.102:3300/0 conn(0x7f815c159a70 0x7f815c0cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 >> v1:172.21.15.102:6789/0 conn(0x7f815c1603b0 legacy=0x7f815c0d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815c0b3470 con 0x7f815c159e50 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f8144001680 con 0x7f815c159e50 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 >> v1:172.21.15.90:6789/0 conn(0x7f815c159e50 legacy=0x7f815c15fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81400039c0 0x7f8140003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 0x7f8140004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.921 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140005220 0x7f814000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f815c0cfc50 con 0x7f8140004810 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f814000c150 con 0x7f8140005220 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f814000c350 con 0x7f81400039c0 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.920+0000 7f8153e00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 0x7f8140004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815b400640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140005220 0x7f814000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.922 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815aa00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81400039c0 0x7f8140003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 -- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81400039c0 msgr2=0x7f8140003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81400039c0 0x7f8140003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 -- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140005220 msgr2=0x7f814000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140005220 0x7f814000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 -- 172.21.15.90:0/159989255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8140003530 con 0x7f8140004810 2024-03-31T15:53:34.923 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815b400640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140005220 0x7f814000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815b400640 1 -- 172.21.15.90:0/159989255 reap_dead start 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815aa00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81400039c0 0x7f8140003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8153e00640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 0x7f8140004c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f814400c260 tx=0x7f814401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3e456431859dec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8144007cf0 con 0x7f8140004810 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81440a9020 con 0x7f8140004810 2024-03-31T15:53:34.924 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f815a000640 1 -- 172.21.15.90:0/159989255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8144005b00 con 0x7f8140004810 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 -- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 msgr2=0x7f8140004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 0x7f8140004c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f814400c260 tx=0x7f814401cbf0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 -- 172.21.15.90:0/159989255 shutdown_connections 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 --2- 172.21.15.90:0/159989255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8140004810 0x7f8140004c90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 -- 172.21.15.90:0/159989255 >> 172.21.15.90:0/159989255 conn(0x7f815c0b3d10 msgr2=0x7f815c0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:34.925 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 -- 172.21.15.90:0/159989255 shutdown_connections 2024-03-31T15:53:34.926 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.924+0000 7f8161800640 1 -- 172.21.15.90:0/159989255 wait complete. 2024-03-31T15:53:34.932 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 Processor -- start 2024-03-31T15:53:34.932 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- start start 2024-03-31T15:53:34.932 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140004810 0x7f815c15e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.933 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 0x7f815c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.933 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.933 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f815c0d0c80 con 0x7f815c147fb0 2024-03-31T15:53:34.933 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f815c0d0b00 con 0x7f8140004810 2024-03-31T15:53:34.933 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f815c15d870 con 0x7f815c0cb390 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53402/0 (socket says 172.21.15.90:53402) 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 -- 172.21.15.90:0/1656934886 learned_addr learned my addr 172.21.15.90:0/1656934886 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815aa00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140004810 0x7f815c15e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8153e00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 0x7f815c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.934 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 -- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 msgr2=0x7f815c0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 0x7f815c0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 -- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140004810 msgr2=0x7f815c15e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140004810 0x7f815c15e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 -- 172.21.15.90:0/1656934886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815c1465c0 con 0x7f815c147fb0 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8153e00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 0x7f815c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:34.935 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f815b400640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f815c0d1fe0 tx=0x7f815400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154003b80 con 0x7f815c147fb0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8154017070 con 0x7f815c147fb0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154005070 con 0x7f815c147fb0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815c148ce0 con 0x7f815c147fb0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.928+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815c1492a0 con 0x7f815c147fb0 2024-03-31T15:53:34.936 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.932+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815c068f20 con 0x7f815c147fb0 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.932+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) ==== 87931+0+0 (secure 0 0 0) 0x7f8154005210 con 0x7f815c147fb0 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.932+0000 7f8152a00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 0x7f813806a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.932+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3071+0+0 (secure 0 0 0) 0x7f8154085bc0 con 0x7f815c147fb0 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.932+0000 7f815aa00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 0x7f813806a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.936+0000 7f815aa00640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 0x7f813806a7b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f815c06a3f0 tx=0x7f814c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:34.937 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:34.940+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8154056d20 con 0x7f815c147fb0 2024-03-31T15:53:35.055 INFO:tasks.ceph.osd.6.smithi102.stderr:2024-03-31T15:53:35.060+0000 7f4929e00640 -1 osd.6 0 waiting for initial osdmap 2024-03-31T15:53:35.058 INFO:tasks.ceph.osd.4.smithi102.stderr:2024-03-31T15:53:35.064+0000 7f643e200640 -1 osd.4 0 waiting for initial osdmap 2024-03-31T15:53:35.084 INFO:tasks.ceph.osd.6.smithi102.stderr:2024-03-31T15:53:35.092+0000 7f491fe00640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:35.085 INFO:tasks.ceph.osd.5.smithi102.stderr:2024-03-31T15:53:35.092+0000 7fe9a2200640 -1 osd.5 0 waiting for initial osdmap 2024-03-31T15:53:35.091 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f815c067560 con 0x7f815c147fb0 2024-03-31T15:53:35.091 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8152a00640 1 -- 172.21.15.90:0/1656934886 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+8514 (secure 0 0 0) 0x7f8154056620 con 0x7f815c147fb0 2024-03-31T15:53:35.091 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-31T15:53:35.092 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":14,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:34.271276+0000","last_up_change":"2024-03-31T15:53:34.271276+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 msgr2=0x7f813806a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 0x7f813806a7b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f815c06a3f0 tx=0x7f814c008040 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 msgr2=0x7f815c148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f815c0d1fe0 tx=0x7f815400dc00 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 shutdown_connections 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81380682f0 0x7f813806a7b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f815c147fb0 0x7f815c148430 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f815c0cb390 0x7f815c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 --2- 172.21.15.90:0/1656934886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8140004810 0x7f815c15e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 >> 172.21.15.90:0/1656934886 conn(0x7f815c0b3d10 msgr2=0x7f815c15c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:35.094 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 shutdown_connections 2024-03-31T15:53:35.095 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:35.100+0000 7f8161800640 1 -- 172.21.15.90:0/1656934886 wait complete. 2024-03-31T15:53:35.104 INFO:tasks.ceph.osd.4.smithi102.stderr:2024-03-31T15:53:35.112+0000 7f6433800640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:35.109 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-03-31T15:53:35.116 INFO:tasks.ceph.osd.5.smithi102.stderr:2024-03-31T15:53:35.124+0000 7fe997800640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-31T15:53:41.916 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:42.052 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 Processor -- start 2024-03-31T15:53:42.052 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 -- start start 2024-03-31T15:53:42.053 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab20146e80 0x7fab200b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.053 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 0x7fab200ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.053 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2006f3c0 con 0x7fab20147510 2024-03-31T15:53:42.053 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fab2006f6a0 con 0x7fab20146e80 2024-03-31T15:53:42.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab26800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab200b54b0 con 0x7fab2006de50 2024-03-31T15:53:42.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab20146e80 0x7fab200b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab20146e80 0x7fab200b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53406/0 (socket says 172.21.15.90:53406) 2024-03-31T15:53:42.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 -- 172.21.15.90:0/2764747682 learned_addr learned my addr 172.21.15.90:0/2764747682 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:42.054 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1ea00640 1 --1- 172.21.15.90:0/2764747682 >> v1:172.21.15.90:6789/0 conn(0x7fab20147510 0x7fab2006d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:51022/0 (socket says 172.21.15.90:51022) 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 -- 172.21.15.90:0/2764747682 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 msgr2=0x7fab200ad870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1fe00640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 0x7fab200ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 0x7fab200ad870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 -- 172.21.15.90:0/2764747682 >> v1:172.21.15.90:6789/0 conn(0x7fab20147510 legacy=0x7fab2006d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1f400640 1 -- 172.21.15.90:0/2764747682 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab200b5790 con 0x7fab20146e80 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1e000640 1 -- 172.21.15.90:0/2764747682 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891791888 0 0) 0x7fab2006f3c0 con 0x7fab20147510 2024-03-31T15:53:42.055 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.059+0000 7fab1fe00640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 0x7fab200ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:42.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab1f400640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab20146e80 0x7fab200b5f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fab10009e50 tx=0x7fab1001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54f2a74eb9ff295 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab1e000640 1 -- 172.21.15.90:0/2764747682 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100a58f0 con 0x7fab20146e80 2024-03-31T15:53:42.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab1e000640 1 -- 172.21.15.90:0/2764747682 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab100a5a90 con 0x7fab20146e80 2024-03-31T15:53:42.057 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab1e000640 1 -- 172.21.15.90:0/2764747682 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100a5e60 con 0x7fab20146e80 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 -- 172.21.15.90:0/2764747682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab20146e80 msgr2=0x7fab200b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab20146e80 0x7fab200b5f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fab10009e50 tx=0x7fab1001ca00 comp rx=0 tx=0).stop 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 -- 172.21.15.90:0/2764747682 shutdown_connections 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab2006de50 0x7fab200ad870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 --2- 172.21.15.90:0/2764747682 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab20146e80 0x7fab200b5f90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 -- 172.21.15.90:0/2764747682 >> 172.21.15.90:0/2764747682 conn(0x7fab200724c0 msgr2=0x7fab200728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 -- 172.21.15.90:0/2764747682 shutdown_connections 2024-03-31T15:53:42.058 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.063+0000 7fab26800640 1 -- 172.21.15.90:0/2764747682 wait complete. 2024-03-31T15:53:42.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 Processor -- start 2024-03-31T15:53:42.059 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- start start 2024-03-31T15:53:42.062 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 0x7fab2013b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 0x7fab2013bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fab2016afb0 con 0x7fab2006de50 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab2016ae30 con 0x7fab20147510 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab2016b130 con 0x7fab20146e80 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53412/0 (socket says 172.21.15.90:53412) 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 -- 172.21.15.90:0/1672385753 learned_addr learned my addr 172.21.15.90:0/1672385753 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1ea00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 0x7fab2013b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1fe00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 0x7fab2013bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 -- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 msgr2=0x7fab2013b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.063 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 0x7fab2013b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 -- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 msgr2=0x7fab2013bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 0x7fab2013bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 -- 172.21.15.90:0/1672385753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab20140fa0 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1fe00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 0x7fab2013bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1ea00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 0x7fab2013b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab1f400640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab100af040 tx=0x7fab100197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100bc020 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab100b8470 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab20141280 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100a58f0 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab20141840 con 0x7fab2006de50 2024-03-31T15:53:42.064 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.067+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab20068f20 con 0x7fab2006de50 2024-03-31T15:53:42.065 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.071+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fab100bf020 con 0x7fab2006de50 2024-03-31T15:53:42.065 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.071+0000 7fab17400640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 0x7fab0006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.065 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.071+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fab10121cb0 con 0x7fab2006de50 2024-03-31T15:53:42.065 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.071+0000 7fab1ea00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 0x7fab0006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.065 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.071+0000 7fab1ea00640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 0x7fab0006a840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab2013d910 tx=0x7fab0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.069 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.075+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fab100c6030 con 0x7fab2006de50 2024-03-31T15:53:42.230 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.235+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab200b4ce0 con 0x7fab2006de50 2024-03-31T15:53:42.231 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab17400640 1 -- 172.21.15.90:0/1672385753 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7fab10016620 con 0x7fab2006de50 2024-03-31T15:53:42.231 INFO:teuthology.misc.health.smithi090.stdout: 2024-03-31T15:53:42.231 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":17,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:37.686469+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:42.233 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 msgr2=0x7fab0006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.233 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 0x7fab0006a840 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab2013d910 tx=0x7fab0c009040 comp rx=0 tx=0).stop 2024-03-31T15:53:42.233 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 msgr2=0x7fab2013e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.233 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fab100af040 tx=0x7fab100197a0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.233 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 shutdown_connections 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab00068380 0x7fab0006a840 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab20147510 0x7fab2013bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab20146e80 0x7fab2013b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 --2- 172.21.15.90:0/1672385753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab2006de50 0x7fab2013e520 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 >> 172.21.15.90:0/1672385753 conn(0x7fab200724c0 msgr2=0x7fab2006d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 shutdown_connections 2024-03-31T15:53:42.234 INFO:teuthology.misc.health.smithi090.stderr:2024-03-31T15:53:42.239+0000 7fab26800640 1 -- 172.21.15.90:0/1672385753 wait complete. 2024-03-31T15:53:42.249 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-31T15:53:42.249 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-31T15:53:42.250 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:53:42.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:53:42.253 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:53:42.253 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:53:42.254 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:53:42.257 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:53:42.257 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:53:42.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:53:42.260 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:53:42.260 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:53:42.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:53:42.264 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:53:42.264 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-31T15:53:42.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:42.366 INFO:tasks.ceph.mds.b.smithi102.stdout:starting mds.b at 2024-03-31T15:53:42.366 INFO:tasks.ceph.mds.d.smithi102.stdout:starting mds.d at 2024-03-31T15:53:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.407+0000 7f98b7e00640 1 Processor -- start 2024-03-31T15:53:42.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.407+0000 7f98b7e00640 1 -- start start 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 0x7f98b80ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98b80d01a0 0x7f98b8072d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b7e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b80d06f0 con 0x7f98b80b5f00 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b7e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f98b80d09d0 con 0x7f98b80d01a0 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f98b80d0cb0 con 0x7f98b815a480 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98b80d01a0 0x7f98b8072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98b80d01a0 0x7f98b8072d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53422/0 (socket says 172.21.15.90:53422) 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 -- 172.21.15.90:0/332574362 learned_addr learned my addr 172.21.15.90:0/332574362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:42.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6a00640 1 --2- 172.21.15.90:0/332574362 >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 0x7f98b80ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 -- 172.21.15.90:0/332574362 >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 msgr2=0x7f98b80ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 --2- 172.21.15.90:0/332574362 >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 0x7f98b80ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 -- 172.21.15.90:0/332574362 >> v1:172.21.15.90:6789/0 conn(0x7f98b80b5f00 legacy=0x7f98b806b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6000640 1 -- 172.21.15.90:0/332574362 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b80d0f90 con 0x7f98b80d01a0 2024-03-31T15:53:42.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.411+0000 7f98b6a00640 1 --2- 172.21.15.90:0/332574362 >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 0x7f98b80ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:42.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b6000640 1 --2- 172.21.15.90:0/332574362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80d01a0 0x7f98b8072d30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98b8071160 tx=0x7f98a401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf614e101d49e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b5600640 1 -- 172.21.15.90:0/332574362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40b0070 con 0x7f98b80d01a0 2024-03-31T15:53:42.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b5600640 1 -- 172.21.15.90:0/332574362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a40ab600 con 0x7f98b80d01a0 2024-03-31T15:53:42.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b5600640 1 -- 172.21.15.90:0/332574362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40ab9d0 con 0x7f98b80d01a0 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- 172.21.15.90:0/332574362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80d01a0 msgr2=0x7f98b8072d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 --2- 172.21.15.90:0/332574362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80d01a0 0x7f98b8072d30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98b8071160 tx=0x7f98a401c490 comp rx=0 tx=0).stop 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- 172.21.15.90:0/332574362 shutdown_connections 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 --2- 172.21.15.90:0/332574362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80d01a0 0x7f98b8072d30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 --2- 172.21.15.90:0/332574362 >> v2:172.21.15.102:3300/0 conn(0x7f98b815a480 0x7f98b80ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- 172.21.15.90:0/332574362 >> 172.21.15.90:0/332574362 conn(0x7f98b80b5130 msgr2=0x7f98b80b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:42.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- 172.21.15.90:0/332574362 shutdown_connections 2024-03-31T15:53:42.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- 172.21.15.90:0/332574362 wait complete. 2024-03-31T15:53:42.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 Processor -- start 2024-03-31T15:53:42.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 -- start start 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.415+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98b80d01a0 0x7f98b814d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 0x7f98b814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8169510 con 0x7f98b80b5f00 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98b806c700 con 0x7f98b815a480 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f98b8169690 con 0x7f98b80d01a0 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53428/0 (socket says 172.21.15.90:53428) 2024-03-31T15:53:42.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 -- 172.21.15.90:0/948343761 learned_addr learned my addr 172.21.15.90:0/948343761 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6000640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98b80d01a0 0x7f98b814d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7400640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 0x7f98b814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 -- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98b80d01a0 msgr2=0x7f98b814d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98b80d01a0 0x7f98b814d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 -- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 msgr2=0x7f98b814ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 0x7f98b814ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 -- 172.21.15.90:0/948343761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b8073040 con 0x7f98b80b5f00 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7400640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 0x7f98b814ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b6a00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98a800d800 tx=0x7f98a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a800b6c0 con 0x7f98b80b5f00 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a8019070 con 0x7f98b80b5f00 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b815d040 con 0x7f98b80b5f00 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a800baa0 con 0x7f98b80b5f00 2024-03-31T15:53:42.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b8152680 con 0x7f98b80b5f00 2024-03-31T15:53:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b8068d00 con 0x7f98b80b5f00 2024-03-31T15:53:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98a800bc40 con 0x7f98b80b5f00 2024-03-31T15:53:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.419+0000 7f98aea00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 0x7f989806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:42.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.423+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f98a8086ab0 con 0x7f98b80b5f00 2024-03-31T15:53:42.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.423+0000 7f98b6000640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 0x7f989806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:42.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.423+0000 7f98b6000640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 0x7f989806a840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f98a40ad000 tx=0x7f98a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:42.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.427+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f98a8057320 con 0x7f98b80b5f00 2024-03-31T15:53:42.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.583+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98b814cf50 con 0x7f98b80b5f00 2024-03-31T15:53:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:42.578 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:37.686469+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.583+0000 7f98aea00640 1 -- 172.21.15.90:0/948343761 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7f98a8056c20 con 0x7f98b80b5f00 2024-03-31T15:53:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 msgr2=0x7f989806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 0x7f989806a840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f98a40ad000 tx=0x7f98a4008040 comp rx=0 tx=0).stop 2024-03-31T15:53:42.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 msgr2=0x7f98b81fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98a800d800 tx=0x7f98a800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 shutdown_connections 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9898068380 0x7f989806a840 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98b815a480 0x7f98b814ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98b80d01a0 0x7f98b814d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 --2- 172.21.15.90:0/948343761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98b80b5f00 0x7f98b81fb2b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 >> 172.21.15.90:0/948343761 conn(0x7f98b80b5130 msgr2=0x7f98b81597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 shutdown_connections 2024-03-31T15:53:42.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:42.587+0000 7f98b7e00640 1 -- 172.21.15.90:0/948343761 wait complete. 2024-03-31T15:53:42.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:53:42.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:53:43.047 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-31T15:53:43.058 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-31T15:53:43.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.079+0000 7ff486e00640 1 Processor -- start 2024-03-31T15:53:43.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.079+0000 7ff486e00640 1 -- start start 2024-03-31T15:53:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff486e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff48015e4e0 0x7ff48006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff486e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff4800cdf20 0x7ff48006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff486e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48015e8c0 con 0x7ff48006ce50 2024-03-31T15:53:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff486e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff4800d0ff0 con 0x7ff48015e4e0 2024-03-31T15:53:43.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff486e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff4800d12d0 con 0x7ff4800cdf20 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff48015e4e0 0x7ff48006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff48006ce50 0x7ff4800ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47438/0 (socket says 172.21.15.90:47438) 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47f400640 1 -- 172.21.15.90:0/1965420209 learned_addr learned my addr 172.21.15.90:0/1965420209 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff484e00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff4800cdf20 0x7ff48006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47fe00640 1 -- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff4800cdf20 msgr2=0x7ff48006d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47fe00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff4800cdf20 0x7ff48006d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47fe00640 1 -- 172.21.15.90:0/1965420209 >> v1:172.21.15.90:6789/0 conn(0x7ff48006ce50 legacy=0x7ff4800ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47fe00640 1 -- 172.21.15.90:0/1965420209 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4800cc710 con 0x7ff48015e4e0 2024-03-31T15:53:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.083+0000 7ff47ea00640 1 -- 172.21.15.90:0/1965420209 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697685480 0 0) 0x7ff48015e8c0 con 0x7ff48006ce50 2024-03-31T15:53:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff47fe00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff48015e4e0 0x7ff48006c910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4700097f0 tx=0x7ff47001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d45baab24b024f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff47ea00640 1 -- 172.21.15.90:0/1965420209 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4700b0070 con 0x7ff48015e4e0 2024-03-31T15:53:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff47ea00640 1 -- 172.21.15.90:0/1965420209 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4700ab600 con 0x7ff48015e4e0 2024-03-31T15:53:43.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff47ea00640 1 -- 172.21.15.90:0/1965420209 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4700ab9d0 con 0x7ff48015e4e0 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff48015e4e0 msgr2=0x7ff48006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff48015e4e0 0x7ff48006c910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4700097f0 tx=0x7ff47001c410 comp rx=0 tx=0).stop 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- 172.21.15.90:0/1965420209 shutdown_connections 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff4800cdf20 0x7ff48006d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 --2- 172.21.15.90:0/1965420209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff48015e4e0 0x7ff48006c910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- 172.21.15.90:0/1965420209 >> 172.21.15.90:0/1965420209 conn(0x7ff4800d1f20 msgr2=0x7ff4800d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- 172.21.15.90:0/1965420209 shutdown_connections 2024-03-31T15:53:43.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- 172.21.15.90:0/1965420209 wait complete. 2024-03-31T15:53:43.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 Processor -- start 2024-03-31T15:53:43.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.087+0000 7ff486e00640 1 -- start start 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 0x7ff480156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff48015e4e0 0x7ff4801571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff48016b1b0 con 0x7ff48006ce50 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff48016b030 con 0x7ff4800cdf20 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff48016b330 con 0x7ff48015e4e0 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33174/0 (socket says 172.21.15.90:33174) 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 -- 172.21.15.90:0/1581498478 learned_addr learned my addr 172.21.15.90:0/1581498478 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 -- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff48015e4e0 msgr2=0x7ff4801571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff48015e4e0 0x7ff4801571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 -- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 msgr2=0x7ff480156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47f400640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 0x7ff480156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 0x7ff480156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 -- 172.21.15.90:0/1581498478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4800d2b90 con 0x7ff48006ce50 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47f400640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 0x7ff480156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff47fe00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4700ae040 tx=0x7ff47001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4700b0050 con 0x7ff48006ce50 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4700b1040 con 0x7ff48006ce50 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470006e20 con 0x7ff48006ce50 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff480068d00 con 0x7ff48006ce50 2024-03-31T15:53:43.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff480155930 con 0x7ff48006ce50 2024-03-31T15:53:43.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4800685f0 con 0x7ff48006ce50 2024-03-31T15:53:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.091+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff4700be030 con 0x7ff48006ce50 2024-03-31T15:53:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.095+0000 7ff477e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 0x7ff46006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.095+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff470120210 con 0x7ff48006ce50 2024-03-31T15:53:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.095+0000 7ff47f400640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 0x7ff46006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.095+0000 7ff47f400640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 0x7ff46006a870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4800c6540 tx=0x7ff46c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.099+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff4700c5020 con 0x7ff48006ce50 2024-03-31T15:53:43.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.255+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4800d1a90 con 0x7ff48006ce50 2024-03-31T15:53:43.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.255+0000 7ff477e00640 1 -- 172.21.15.90:0/1581498478 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7ff470016090 con 0x7ff48006ce50 2024-03-31T15:53:43.250 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 msgr2=0x7ff46006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 0x7ff46006a870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4800c6540 tx=0x7ff46c009040 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 msgr2=0x7ff4800cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4700ae040 tx=0x7ff47001ac50 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 shutdown_connections 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff4600683b0 0x7ff46006a870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff48015e4e0 0x7ff4801571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4800cdf20 0x7ff480156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 --2- 172.21.15.90:0/1581498478 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff48006ce50 0x7ff4800cb720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 >> 172.21.15.90:0/1581498478 conn(0x7ff4800d1f20 msgr2=0x7ff480070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 shutdown_connections 2024-03-31T15:53:43.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.259+0000 7ff486e00640 1 -- 172.21.15.90:0/1581498478 wait complete. 2024-03-31T15:53:43.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:53:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 Processor -- start 2024-03-31T15:53:43.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 -- start start 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c06b5f0 0x7f7e8c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7e8c165e90 0x7f7e8c168280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e8c0b0470 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7e8c0b0750 con 0x7f7e8c165e90 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8c0b0a30 con 0x7f7e8c06b5f0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7e8c15e1c0 0x7f7e8c0b5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47442/0 (socket says 172.21.15.90:47442) 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 -- 172.21.15.90:0/1847460899 learned_addr learned my addr 172.21.15.90:0/1847460899 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302646895 0 0) 0x7f7e8c0b0470 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e74003530 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e90c00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c06b5f0 0x7f7e8c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92000640 1 --2- 172.21.15.90:0/1847460899 >> v2:172.21.15.102:3300/0 conn(0x7f7e8c165e90 0x7f7e8c168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 203154689 0 0) 0x7f7e74003530 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c06b5f0 msgr2=0x7f7e8c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c06b5f0 0x7f7e8c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 >> v2:172.21.15.102:3300/0 conn(0x7f7e8c165e90 msgr2=0x7f7e8c168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 --2- 172.21.15.90:0/1847460899 >> v2:172.21.15.102:3300/0 conn(0x7f7e8c165e90 0x7f7e8c168280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c0b0d10 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7e80001680 con 0x7f7e8c15e1c0 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 >> v1:172.21.15.90:6789/0 conn(0x7f7e8c15e1c0 legacy=0x7f7e8c0b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e74004810 0x7f7e74004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e740051d0 0x7f7e7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c0b0750 con 0x7f7e740039c0 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c0b0a30 con 0x7f7e740051d0 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7400bf90 con 0x7f7e74004810 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e90c00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c06b5f0 0x7f7e8c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:43.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92000640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e740051d0 0x7f7e7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e90c00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e74004810 0x7f7e74004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 -- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e74004810 msgr2=0x7f7e74004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e74004810 0x7f7e74004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 -- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e740051d0 msgr2=0x7f7e7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e740051d0 0x7f7e7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e91600640 1 -- 172.21.15.90:0/1847460899 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e74003530 con 0x7f7e740039c0 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e90c00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e74004810 0x7f7e74004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:43.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.415+0000 7f7e92000640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e740051d0 0x7f7e7400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e91600640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e74003de0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000c160 tx=0x7f7e8001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cdbc7d9cef0dac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80007c90 con 0x7f7e740039c0 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e800a9020 con 0x7f7e740039c0 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e8be00640 1 -- 172.21.15.90:0/1847460899 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80005ae0 con 0x7f7e740039c0 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 -- 172.21.15.90:0/1847460899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 msgr2=0x7f7e74003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 --2- 172.21.15.90:0/1847460899 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e74003de0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000c160 tx=0x7f7e8001cbe0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92000640 1 -- 172.21.15.90:0/1847460899 reap_dead start 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 -- 172.21.15.90:0/1847460899 shutdown_connections 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 -- 172.21.15.90:0/1847460899 >> 172.21.15.90:0/1847460899 conn(0x7f7e8c072d80 msgr2=0x7f7e8c073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 -- 172.21.15.90:0/1847460899 shutdown_connections 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.419+0000 7f7e92a00640 1 -- 172.21.15.90:0/1847460899 wait complete. 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 Processor -- start 2024-03-31T15:53:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- start start 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c157f50 0x7f7e8c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e8c0ae830 0x7f7e8c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c06bf60 con 0x7f7e740039c0 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8c06bde0 con 0x7f7e8c0ae830 2024-03-31T15:53:43.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7e8c15e5a0 con 0x7f7e8c157f50 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33186/0 (socket says 172.21.15.90:33186) 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 -- 172.21.15.90:0/2835559093 learned_addr learned my addr 172.21.15.90:0/2835559093 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e90c00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c157f50 0x7f7e8c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92000640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e8c0ae830 0x7f7e8c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 -- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c157f50 msgr2=0x7f7e8c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c157f50 0x7f7e8c0c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 -- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e8c0ae830 msgr2=0x7f7e8c15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e8c0ae830 0x7f7e8c15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 -- 172.21.15.90:0/2835559093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8c0ad790 con 0x7f7e740039c0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e91600640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e8001ca10 tx=0x7f7e74005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e800049f0 con 0x7f7e740039c0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e80004b90 con 0x7f7e740039c0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e800070d0 con 0x7f7e740039c0 2024-03-31T15:53:43.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8c0ada70 con 0x7f7e740039c0 2024-03-31T15:53:43.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8c0ae030 con 0x7f7e740039c0 2024-03-31T15:53:43.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.423+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e8c068d00 con 0x7f7e740039c0 2024-03-31T15:53:43.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.427+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7e80007270 con 0x7f7e740039c0 2024-03-31T15:53:43.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.427+0000 7f7e89600640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 0x7f7e6c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.427+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7e8011fc10 con 0x7f7e740039c0 2024-03-31T15:53:43.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.427+0000 7f7e90c00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 0x7f7e6c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.427+0000 7f7e90c00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 0x7f7e6c06a870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0af260 tx=0x7f7e78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.431+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7e800168f0 con 0x7f7e740039c0 2024-03-31T15:53:43.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.619+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e8c0b2580 con 0x7f7e740039c0 2024-03-31T15:53:43.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.619+0000 7f7e89600640 1 -- 172.21.15.90:0/2835559093 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3864 (secure 0 0 0) 0x7f7e800f05b0 con 0x7f7e740039c0 2024-03-31T15:53:43.614 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:43.614 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-31T15:53:43.615 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 3 2024-03-31T15:53:43.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 msgr2=0x7f7e6c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 0x7f7e6c06a870 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c0af260 tx=0x7f7e78008040 comp rx=0 tx=0).stop 2024-03-31T15:53:43.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 msgr2=0x7f7e8c1616b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7e8001ca10 tx=0x7f7e74005840 comp rx=0 tx=0).stop 2024-03-31T15:53:43.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 shutdown_connections 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7e6c0683b0 0x7f7e6c06a870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7e8c0ae830 0x7f7e8c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7e8c157f50 0x7f7e8c0c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 --2- 172.21.15.90:0/2835559093 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7e740039c0 0x7f7e8c1616b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 >> 172.21.15.90:0/2835559093 conn(0x7f7e8c072d80 msgr2=0x7f7e8c163740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 shutdown_connections 2024-03-31T15:53:43.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.623+0000 7f7e92a00640 1 -- 172.21.15.90:0/2835559093 wait complete. 2024-03-31T15:53:43.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.775+0000 7fa0eee00640 1 Processor -- start 2024-03-31T15:53:43.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.775+0000 7fa0eee00640 1 -- start start 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0eee00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa0e815c120 0x7fa0e815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0eee00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 0x7fa0e80ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0eee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0e80d2b10 con 0x7fa0e80d2680 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0eee00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa0e8160bf0 con 0x7fa0e815c120 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0eee00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e8160350 con 0x7fa0e80cca30 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0df400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa0e80d2680 0x7fa0e80b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47444/0 (socket says 172.21.15.90:47444) 2024-03-31T15:53:43.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0df400640 1 -- 172.21.15.90:0/378671940 learned_addr learned my addr 172.21.15.90:0/378671940 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0ece00640 1 --2- 172.21.15.90:0/378671940 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 0x7fa0e80ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0e7e00640 1 --2- 172.21.15.90:0/378671940 >> v2:172.21.15.102:3300/0 conn(0x7fa0e815c120 0x7fa0e815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750008155 0 0) 0x7fa0e80d2b10 con 0x7fa0e80d2680 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0cc003530 con 0x7fa0e80d2680 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0ece00640 1 -- 172.21.15.90:0/378671940 >> v2:172.21.15.102:3300/0 conn(0x7fa0e815c120 msgr2=0x7fa0e815fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:43.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0ece00640 1 --2- 172.21.15.90:0/378671940 >> v2:172.21.15.102:3300/0 conn(0x7fa0e815c120 0x7fa0e815fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0ece00640 1 -- 172.21.15.90:0/378671940 >> v1:172.21.15.90:6789/0 conn(0x7fa0e80d2680 legacy=0x7fa0e80b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2060454890 0 0) 0x7fa0cc003530 con 0x7fa0e80d2680 2024-03-31T15:53:43.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.779+0000 7fa0ece00640 1 -- 172.21.15.90:0/378671940 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e8160630 con 0x7fa0e80cca30 2024-03-31T15:53:43.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0ece00640 1 --2- 172.21.15.90:0/378671940 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 0x7fa0e80ccea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009d90 tx=0x7fa0e001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beafb02d05f30671 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0001680 con 0x7fa0e80cca30 2024-03-31T15:53:43.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e00a6500 con 0x7fa0e80cca30 2024-03-31T15:53:43.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0e7400640 1 -- 172.21.15.90:0/378671940 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00a68d0 con 0x7fa0e80cca30 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- 172.21.15.90:0/378671940 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 msgr2=0x7fa0e80ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- 172.21.15.90:0/378671940 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 0x7fa0e80ccea0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009d90 tx=0x7fa0e001c800 comp rx=0 tx=0).stop 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- 172.21.15.90:0/378671940 shutdown_connections 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- 172.21.15.90:0/378671940 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e80cca30 0x7fa0e80ccea0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- 172.21.15.90:0/378671940 >> v2:172.21.15.102:3300/0 conn(0x7fa0e815c120 0x7fa0e815fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- 172.21.15.90:0/378671940 >> 172.21.15.90:0/378671940 conn(0x7fa0e809aa40 msgr2=0x7fa0e809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- 172.21.15.90:0/378671940 shutdown_connections 2024-03-31T15:53:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- 172.21.15.90:0/378671940 wait complete. 2024-03-31T15:53:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 Processor -- start 2024-03-31T15:53:43.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- start start 2024-03-31T15:53:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 0x7fa0e8152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e815c120 0x7fa0e80cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80b2f70 con 0x7fa0e80d2680 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80b2df0 con 0x7fa0e80cca30 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.783+0000 7fa0eee00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e8169040 con 0x7fa0e815c120 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 0x7fa0e8152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0ece00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e815c120 0x7fa0e80cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33192/0 (socket says 172.21.15.90:33192) 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 -- 172.21.15.90:0/3166063001 learned_addr learned my addr 172.21.15.90:0/3166063001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 0x7fa0e8152640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:36742/0 (socket says 172.21.15.90:36742) 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 -- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e815c120 msgr2=0x7fa0e80cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e815c120 0x7fa0e80cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 -- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 msgr2=0x7fa0e8152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 0x7fa0e8152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 -- 172.21.15.90:0/3166063001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e80c6b40 con 0x7fa0e80d2680 2024-03-31T15:53:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0df400640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c550 tx=0x7fa0d000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0017070 con 0x7fa0e80d2680 2024-03-31T15:53:43.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d0005cf0 con 0x7fa0e80d2680 2024-03-31T15:53:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00072b0 con 0x7fa0e80d2680 2024-03-31T15:53:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e80c6e20 con 0x7fa0e80d2680 2024-03-31T15:53:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e80c73e0 con 0x7fa0e80d2680 2024-03-31T15:53:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e80686d0 con 0x7fa0e80d2680 2024-03-31T15:53:43.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa0d001a070 con 0x7fa0e80d2680 2024-03-31T15:53:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.787+0000 7fa0e4c00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 0x7fa0c406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.791+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa0d0085ac0 con 0x7fa0e80d2680 2024-03-31T15:53:43.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.791+0000 7fa0e7e00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 0x7fa0c406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:43.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.791+0000 7fa0e7e00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 0x7fa0c406a870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009ac0 tx=0x7fa0d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:43.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.791+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa0d0055ce0 con 0x7fa0e80d2680 2024-03-31T15:53:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.947+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0e80c75b0 con 0x7fa0e80d2680 2024-03-31T15:53:43.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0e4c00640 1 -- 172.21.15.90:0/3166063001 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12848 (secure 0 0 0) 0x7fa0d0059fe0 con 0x7fa0e80d2680 2024-03-31T15:53:43.944 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:43.944 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:37.686469+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 msgr2=0x7fa0c406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 0x7fa0c406a870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009ac0 tx=0x7fa0d8009190 comp rx=0 tx=0).stop 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 msgr2=0x7fa0e80cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c550 tx=0x7fa0d000ca10 comp rx=0 tx=0).stop 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 shutdown_connections 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0c40683b0 0x7fa0c406a870 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa0e815c120 0x7fa0e80cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e80d2680 0x7fa0e80cd550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 --2- 172.21.15.90:0/3166063001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0e80cca30 0x7fa0e8152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:43.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 >> 172.21.15.90:0/3166063001 conn(0x7fa0e809aa40 msgr2=0x7fa0e8070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:43.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 shutdown_connections 2024-03-31T15:53:43.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:43.951+0000 7fa0eee00640 1 -- 172.21.15.90:0/3166063001 wait complete. 2024-03-31T15:53:43.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:53:43.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:53:44.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.107+0000 7f4b0c000640 1 Processor -- start 2024-03-31T15:53:44.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.107+0000 7f4b0c000640 1 -- start start 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b0c000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b040d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b0c000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0415ea70 0x7f4b0406d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b0c000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b040d0390 con 0x7f4b0406daf0 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b0c000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4b040d0630 con 0x7f4b0415ea70 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b0c000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4b040d0910 con 0x7f4b0415e690 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0415ea70 0x7f4b0406d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0415ea70 0x7f4b0406d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33204/0 (socket says 172.21.15.90:33204) 2024-03-31T15:53:44.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 -- 172.21.15.90:0/1233420308 learned_addr learned my addr 172.21.15.90:0/1233420308 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 -- 172.21.15.90:0/1233420308 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 msgr2=0x7f4b040d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b09600640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b040d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b040d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 -- 172.21.15.90:0/1233420308 >> v1:172.21.15.102:6789/0 conn(0x7f4b0406daf0 legacy=0x7f4b040ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b08c00640 1 -- 172.21.15.90:0/1233420308 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b040d0bf0 con 0x7f4b0415ea70 2024-03-31T15:53:44.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.111+0000 7f4b09600640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b040d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:44.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b08c00640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0415ea70 0x7f4b0406d5b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009870 tx=0x7f4af001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7315a0695558194a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b03e00640 1 -- 172.21.15.90:0/1233420308 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00ab620 con 0x7f4b0415ea70 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b03e00640 1 -- 172.21.15.90:0/1233420308 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af00b0070 con 0x7f4b0415ea70 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b03e00640 1 -- 172.21.15.90:0/1233420308 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00ab9e0 con 0x7f4b0415ea70 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- 172.21.15.90:0/1233420308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0415ea70 msgr2=0x7f4b0406d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0415ea70 0x7f4b0406d5b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4af0009870 tx=0x7f4af001c460 comp rx=0 tx=0).stop 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- 172.21.15.90:0/1233420308 shutdown_connections 2024-03-31T15:53:44.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0415ea70 0x7f4b0406d5b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- 172.21.15.90:0/1233420308 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b040d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- 172.21.15.90:0/1233420308 >> 172.21.15.90:0/1233420308 conn(0x7f4b040d1f20 msgr2=0x7f4b040d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- 172.21.15.90:0/1233420308 shutdown_connections 2024-03-31T15:53:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- 172.21.15.90:0/1233420308 wait complete. 2024-03-31T15:53:44.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 Processor -- start 2024-03-31T15:53:44.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- start start 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b04150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b0415ea70 0x7f4b04159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b040cf030 con 0x7f4b0406daf0 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b040ceeb0 con 0x7f4b0415ea70 2024-03-31T15:53:44.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.115+0000 7f4b0c000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4b040ccd80 con 0x7f4b0415e690 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b0a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b0415ea70 0x7f4b04159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33212/0 (socket says 172.21.15.90:33212) 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 -- 172.21.15.90:0/2719405246 learned_addr learned my addr 172.21.15.90:0/2719405246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b08c00640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b04150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 -- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 msgr2=0x7f4b04150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b04150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 -- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b0415ea70 msgr2=0x7f4b04159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b0415ea70 0x7f4b04159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 -- 172.21.15.90:0/2719405246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b040b5a80 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b09600640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c7f0 tx=0x7f4af800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005ba0 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af8005d40 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b040b7460 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8007a50 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b041509d0 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b040685f0 con 0x7f4b0406daf0 2024-03-31T15:53:44.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.119+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4af8007bf0 con 0x7f4b0406daf0 2024-03-31T15:53:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.123+0000 7f4b01600640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 0x7f4ae406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.123+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f4af8086700 con 0x7f4b0406daf0 2024-03-31T15:53:44.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.123+0000 7f4b08c00640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 0x7f4ae406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.123+0000 7f4b08c00640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 0x7f4ae406a870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4af00af000 tx=0x7f4af00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.127+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4af8015020 con 0x7f4b0406daf0 2024-03-31T15:53:44.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4b040d27d0 con 0x7f4b0406daf0 2024-03-31T15:53:44.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b01600640 1 -- 172.21.15.90:0/2719405246 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f4af8056920 con 0x7f4b0406daf0 2024-03-31T15:53:44.277 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:53:44.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 msgr2=0x7f4ae406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 0x7f4ae406a870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4af00af000 tx=0x7f4af00a7000 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 msgr2=0x7f4b0415a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c7f0 tx=0x7f4af800cbf0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 shutdown_connections 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae40683b0 0x7f4ae406a870 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b0415ea70 0x7f4b04159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e690 0x7f4b04150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 --2- 172.21.15.90:0/2719405246 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406daf0 0x7f4b0415a640 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 >> 172.21.15.90:0/2719405246 conn(0x7f4b040d1f20 msgr2=0x7f4b04155150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 shutdown_connections 2024-03-31T15:53:44.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.283+0000 7f4b0c000640 1 -- 172.21.15.90:0/2719405246 wait complete. 2024-03-31T15:53:44.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:53:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 Processor -- start 2024-03-31T15:53:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 -- start start 2024-03-31T15:53:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8fc06d7c0 0x7fd8fc15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc15f780 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8fc15f1c0 con 0x7fd8fc15e1b0 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd902800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd8fc1622b0 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8fc06d7c0 0x7fd8fc15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8fc06d7c0 0x7fd8fc15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33224/0 (socket says 172.21.15.90:33224) 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 -- 172.21.15.90:0/1419522604 learned_addr learned my addr 172.21.15.90:0/1419522604 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8faa00640 1 --1- 172.21.15.90:0/1419522604 >> v1:172.21.15.90:6789/0 conn(0x7fd8fc06c7b0 0x7fd8fc06d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47448/0 (socket says 172.21.15.90:47448) 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 -- 172.21.15.90:0/1419522604 >> v1:172.21.15.102:6789/0 conn(0x7fd8fc15e1b0 legacy=0x7fd8fc162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 -- 172.21.15.90:0/1419522604 >> v1:172.21.15.90:6789/0 conn(0x7fd8fc06c7b0 legacy=0x7fd8fc06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fbe00640 1 -- 172.21.15.90:0/1419522604 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc15f4a0 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.439+0000 7fd8fa000640 1 -- 172.21.15.90:0/1419522604 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 968487820 0 0) 0x7fd8fc15f780 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd8fbe00640 1 --2- 172.21.15.90:0/1419522604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06d7c0 0x7fd8fc15e590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be2a187482ccefd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd8fa000640 1 -- 172.21.15.90:0/1419522604 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40a5830 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd8fa000640 1 -- 172.21.15.90:0/1419522604 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f40a59d0 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd8fa000640 1 -- 172.21.15.90:0/1419522604 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40a5da0 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 -- 172.21.15.90:0/1419522604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06d7c0 msgr2=0x7fd8fc15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 --2- 172.21.15.90:0/1419522604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06d7c0 0x7fd8fc15e590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f401c970 comp rx=0 tx=0).stop 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 -- 172.21.15.90:0/1419522604 shutdown_connections 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 --2- 172.21.15.90:0/1419522604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06d7c0 0x7fd8fc15e590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 -- 172.21.15.90:0/1419522604 >> 172.21.15.90:0/1419522604 conn(0x7fd8fc09aa40 msgr2=0x7fd8fc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 -- 172.21.15.90:0/1419522604 shutdown_connections 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 -- 172.21.15.90:0/1419522604 wait complete. 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.443+0000 7fd902800640 1 Processor -- start 2024-03-31T15:53:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- start start 2024-03-31T15:53:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 0x7fd8fc153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8fc15e1b0 0x7fd8fc154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0ce7a0 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd8fc0d2a90 con 0x7fd8fc15e1b0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd8fc0ce920 con 0x7fd8fc06d7c0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33236/0 (socket says 172.21.15.90:33236) 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 -- 172.21.15.90:0/2629535177 learned_addr learned my addr 172.21.15.90:0/2629535177 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fbe00640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8fc15e1b0 0x7fd8fc154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8faa00640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 0x7fd8fc153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 -- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 msgr2=0x7fd8fc153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 0x7fd8fc153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 -- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8fc15e1b0 msgr2=0x7fd8fc154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8fc15e1b0 0x7fd8fc154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 -- 172.21.15.90:0/2629535177 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8fc155600 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8faa00640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 0x7fd8fc153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:44.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8fb400640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c3c0 tx=0x7fd8ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec017070 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0052d0 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8fc0b0840 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec003840 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8fc0b0e00 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.447+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8fc0686d0 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.451+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8ec01a070 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.451+0000 7fd8f3400640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 0x7fd8dc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.451+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd8ec085850 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.451+0000 7fd8faa00640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 0x7fd8dc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.451+0000 7fd8faa00640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 0x7fd8dc06a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0c59e0 tx=0x7fd8e8007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.455+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd8ec005480 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.611+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd8fc16afa0 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.611+0000 7fd8f3400640 1 -- 172.21.15.90:0/2629535177 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fd8ec055b80 con 0x7fd8fc06c7b0 2024-03-31T15:53:44.605 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:53:44.606 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-31T15:53:44.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 msgr2=0x7fd8dc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 0x7fd8dc06a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0c59e0 tx=0x7fd8e8007c60 comp rx=0 tx=0).stop 2024-03-31T15:53:44.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 msgr2=0x7fd8fc0b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c3c0 tx=0x7fd8ec00c880 comp rx=0 tx=0).stop 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 shutdown_connections 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8dc068380 0x7fd8dc06a840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8fc15e1b0 0x7fd8fc154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8fc06d7c0 0x7fd8fc153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 --2- 172.21.15.90:0/2629535177 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8fc06c7b0 0x7fd8fc0b01d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 >> 172.21.15.90:0/2629535177 conn(0x7fd8fc09aa40 msgr2=0x7fd8fc06f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 shutdown_connections 2024-03-31T15:53:44.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.615+0000 7fd902800640 1 -- 172.21.15.90:0/2629535177 wait complete. 2024-03-31T15:53:44.626 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:53:44.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:53:44.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 Processor -- start 2024-03-31T15:53:44.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 -- start start 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff04015c1f0 0x7ff04015c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04015cd60 con 0x7ff04006ea10 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04015d040 con 0x7ff04015be10 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff046c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff04006eec0 con 0x7ff04015c1f0 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff04015c1f0 0x7ff04015c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff04015c1f0 0x7ff04015c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33248/0 (socket says 172.21.15.90:33248) 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 -- 172.21.15.90:0/1409644768 learned_addr learned my addr 172.21.15.90:0/1409644768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff03f400640 1 --1- 172.21.15.90:0/1409644768 >> v1:172.21.15.90:6789/0 conn(0x7ff04006ea10 0x7ff040160250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47460/0 (socket says 172.21.15.90:47460) 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 -- 172.21.15.90:0/1409644768 >> v1:172.21.15.102:6789/0 conn(0x7ff04015be10 legacy=0x7ff0400cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 -- 172.21.15.90:0/1409644768 >> v1:172.21.15.90:6789/0 conn(0x7ff04006ea10 legacy=0x7ff040160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff03ea00640 1 -- 172.21.15.90:0/1409644768 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035194233 0 0) 0x7ff04015cd60 con 0x7ff04006ea10 2024-03-31T15:53:44.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.775+0000 7ff044c00640 1 -- 172.21.15.90:0/1409644768 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04015d320 con 0x7ff04015c1f0 2024-03-31T15:53:44.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff044c00640 1 --2- 172.21.15.90:0/1409644768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff04015c5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff038009e50 tx=0x7ff03801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a13369f9e82691f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff03ea00640 1 -- 172.21.15.90:0/1409644768 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380a5620 con 0x7ff04015c1f0 2024-03-31T15:53:44.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff03ea00640 1 -- 172.21.15.90:0/1409644768 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0380a57c0 con 0x7ff04015c1f0 2024-03-31T15:53:44.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff03ea00640 1 -- 172.21.15.90:0/1409644768 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380a5b90 con 0x7ff04015c1f0 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- 172.21.15.90:0/1409644768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 msgr2=0x7ff04015c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 --2- 172.21.15.90:0/1409644768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff04015c5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff038009e50 tx=0x7ff03801c960 comp rx=0 tx=0).stop 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- 172.21.15.90:0/1409644768 shutdown_connections 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 --2- 172.21.15.90:0/1409644768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff04015c5f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- 172.21.15.90:0/1409644768 >> 172.21.15.90:0/1409644768 conn(0x7ff0400b1880 msgr2=0x7ff0400b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- 172.21.15.90:0/1409644768 shutdown_connections 2024-03-31T15:53:44.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- 172.21.15.90:0/1409644768 wait complete. 2024-03-31T15:53:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 Processor -- start 2024-03-31T15:53:44.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.779+0000 7ff046c00640 1 -- start start 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff04006ea10 0x7ff040152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04015be10 0x7ff0401529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff04016a170 con 0x7ff04015c1f0 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff04016a2f0 con 0x7ff04015be10 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff04016a470 con 0x7ff04006ea10 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33250/0 (socket says 172.21.15.90:33250) 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 -- 172.21.15.90:0/702562711 learned_addr learned my addr 172.21.15.90:0/702562711 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff03f400640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04015be10 0x7ff0401529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff03fe00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff04006ea10 0x7ff040152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 -- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff04006ea10 msgr2=0x7ff040152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff04006ea10 0x7ff040152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 -- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04015be10 msgr2=0x7ff0401529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04015be10 0x7ff0401529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 -- 172.21.15.90:0/702562711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0400b2a60 con 0x7ff04015c1f0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff044c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0380af040 tx=0x7ff038019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380bc020 con 0x7ff04015c1f0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0380b8670 con 0x7ff04015c1f0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff040156ab0 con 0x7ff04015c1f0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0380b8a60 con 0x7ff04015c1f0 2024-03-31T15:53:44.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0400b3000 con 0x7ff04015c1f0 2024-03-31T15:53:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.783+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0400686d0 con 0x7ff04015c1f0 2024-03-31T15:53:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.787+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0380bf020 con 0x7ff04015c1f0 2024-03-31T15:53:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.787+0000 7ff037e00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 0x7ff02006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:44.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.787+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff038121670 con 0x7ff04015c1f0 2024-03-31T15:53:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.787+0000 7ff03fe00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 0x7ff02006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:44.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.787+0000 7ff03fe00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 0x7ff02006a840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0400c9cb0 tx=0x7ff030009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.791+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff0380c6030 con 0x7ff04015c1f0 2024-03-31T15:53:44.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:44.947+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff04006e6a0 con 0x7ff04015c1f0 2024-03-31T15:53:45.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff037e00640 1 -- 172.21.15.90:0/702562711 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7ff0380165e0 con 0x7ff04015c1f0 2024-03-31T15:53:45.716 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-31T15:53:45.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 msgr2=0x7ff02006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 0x7ff02006a840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0400c9cb0 tx=0x7ff030009060 comp rx=0 tx=0).stop 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 msgr2=0x7ff040155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0380af040 tx=0x7ff038019700 comp rx=0 tx=0).stop 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 shutdown_connections 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff020068380 0x7ff02006a840 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff04015c1f0 0x7ff040155ed0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff04015be10 0x7ff0401529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 --2- 172.21.15.90:0/702562711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff04006ea10 0x7ff040152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.723+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 >> 172.21.15.90:0/702562711 conn(0x7ff0400b1880 msgr2=0x7ff040160250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:45.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.727+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 shutdown_connections 2024-03-31T15:53:45.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.727+0000 7ff046c00640 1 -- 172.21.15.90:0/702562711 wait complete. 2024-03-31T15:53:45.744 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-31T15:53:45.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.891+0000 7f9142600640 1 Processor -- start 2024-03-31T15:53:45.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9142600640 1 -- start start 2024-03-31T15:53:45.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9142600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:45.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9142600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913c0d0390 con 0x7f913c15ea70 2024-03-31T15:53:45.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9142600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f913c0d0630 con 0x7f913c0ce7a0 2024-03-31T15:53:45.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9142600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f913c0d0910 con 0x7f913c15e690 2024-03-31T15:53:45.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9132a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f913c15ea70 0x7f913c06d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47472/0 (socket says 172.21.15.90:47472) 2024-03-31T15:53:45.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f9132a00640 1 -- 172.21.15.90:0/4150377245 learned_addr learned my addr 172.21.15.90:0/4150377245 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:45.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913b400640 1 --2- 172.21.15.90:0/4150377245 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:45.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195126610 0 0) 0x7f913c0d0390 con 0x7f913c15ea70 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9120003530 con 0x7f913c15ea70 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412562864 0 0) 0x7f913c0d0630 con 0x7f913c0ce7a0 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f913c0d0390 con 0x7f913c0ce7a0 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913b400640 1 -- 172.21.15.90:0/4150377245 >> v1:172.21.15.102:6789/0 conn(0x7f913c0ce7a0 legacy=0x7f913c0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913b400640 1 -- 172.21.15.90:0/4150377245 >> v1:172.21.15.90:6789/0 conn(0x7f913c15ea70 legacy=0x7f913c06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.895+0000 7f913b400640 1 -- 172.21.15.90:0/4150377245 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c0d0bf0 con 0x7f913c15e690 2024-03-31T15:53:45.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f913b400640 1 --2- 172.21.15.90:0/4150377245 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0d16d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f912c0097d0 tx=0x7f912c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2919de591e427e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:45.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0ab600 con 0x7f913c15e690 2024-03-31T15:53:45.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0b0070 con 0x7f913c15e690 2024-03-31T15:53:45.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f913aa00640 1 -- 172.21.15.90:0/4150377245 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0ab9a0 con 0x7f913c15e690 2024-03-31T15:53:45.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 -- 172.21.15.90:0/4150377245 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 msgr2=0x7f913c0d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 --2- 172.21.15.90:0/4150377245 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0d16d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f912c0097d0 tx=0x7f912c01c400 comp rx=0 tx=0).stop 2024-03-31T15:53:45.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 -- 172.21.15.90:0/4150377245 shutdown_connections 2024-03-31T15:53:45.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 --2- 172.21.15.90:0/4150377245 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0d16d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 -- 172.21.15.90:0/4150377245 >> 172.21.15.90:0/4150377245 conn(0x7f913c0d1f20 msgr2=0x7f913c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:45.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 -- 172.21.15.90:0/4150377245 shutdown_connections 2024-03-31T15:53:45.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 -- 172.21.15.90:0/4150377245 wait complete. 2024-03-31T15:53:45.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.899+0000 7f9142600640 1 Processor -- start 2024-03-31T15:53:45.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- start start 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f913c15ea70 0x7f913c0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0cefc0 con 0x7f913c0ce7a0 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f913c0b34c0 con 0x7f913c15ea70 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f913c0c9e60 con 0x7f913c15e690 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33266/0 (socket says 172.21.15.90:33266) 2024-03-31T15:53:45.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 -- 172.21.15.90:0/2897973786 learned_addr learned my addr 172.21.15.90:0/2897973786 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 -- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 msgr2=0x7f913c0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 -- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f913c15ea70 msgr2=0x7f913c0ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f913c15ea70 0x7f913c0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 -- 172.21.15.90:0/2897973786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c0cb090 con 0x7f913c0ce7a0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f913b400640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f912c009720 tx=0x7f912c0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c003820 con 0x7f913c0ce7a0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0039c0 con 0x7f913c0ce7a0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913c0cb370 con 0x7f913c0ce7a0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c003d70 con 0x7f913c0ce7a0 2024-03-31T15:53:45.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913c0c9400 con 0x7f913c0ce7a0 2024-03-31T15:53:45.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.903+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f913c0685f0 con 0x7f913c0ce7a0 2024-03-31T15:53:45.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.907+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f912c0c3060 con 0x7f913c0ce7a0 2024-03-31T15:53:45.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.907+0000 7f9133e00640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 0x7f911806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:45.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.907+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f912c1205f0 con 0x7f913c0ce7a0 2024-03-31T15:53:45.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.907+0000 7f9132a00640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 0x7f911806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:45.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.907+0000 7f9132a00640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 0x7f911806a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9124007fd0 tx=0x7f9124007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:45.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:45.911+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f912c0160b0 con 0x7f913c0ce7a0 2024-03-31T15:53:46.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.067+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f913c06c360 con 0x7f913c0ce7a0 2024-03-31T15:53:46.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.727+0000 7f9133e00640 1 -- 172.21.15.90:0/2897973786 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f912c0b2310 con 0x7f913c0ce7a0 2024-03-31T15:53:46.723 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 msgr2=0x7f911806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 0x7f911806a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9124007fd0 tx=0x7f9124007960 comp rx=0 tx=0).stop 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 msgr2=0x7f913c14c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f912c009720 tx=0x7f912c0191a0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 shutdown_connections 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9118068380 0x7f911806a840 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f913c15ea70 0x7f913c0ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f913c15e690 0x7f913c0c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 --2- 172.21.15.90:0/2897973786 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f913c0ce7a0 0x7f913c14c770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 >> 172.21.15.90:0/2897973786 conn(0x7f913c0d1f20 msgr2=0x7f913c06d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 shutdown_connections 2024-03-31T15:53:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.731+0000 7f9142600640 1 -- 172.21.15.90:0/2897973786 wait complete. 2024-03-31T15:53:46.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:53:46.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.903+0000 7fde56400640 1 Processor -- start 2024-03-31T15:53:46.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.903+0000 7fde56400640 1 -- start start 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde56400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fde5015e4d0 0x7fde5006d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde56400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde500d0a30 0x7fde500cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde56400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde500d0f90 con 0x7fde50161e10 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde56400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fde500d1270 con 0x7fde500d0a30 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde56400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fde500d1550 con 0x7fde5015e4d0 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde500d0a30 0x7fde500cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde500d0a30 0x7fde500cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33268/0 (socket says 172.21.15.90:33268) 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 -- 172.21.15.90:0/3710441451 learned_addr learned my addr 172.21.15.90:0/3710441451 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:46.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde4fe00640 1 --2- 172.21.15.90:0/3710441451 >> v2:172.21.15.102:3300/0 conn(0x7fde5015e4d0 0x7fde5006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 -- 172.21.15.90:0/3710441451 >> v2:172.21.15.102:3300/0 conn(0x7fde5015e4d0 msgr2=0x7fde5006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 --2- 172.21.15.90:0/3710441451 >> v2:172.21.15.102:3300/0 conn(0x7fde5015e4d0 0x7fde5006d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 -- 172.21.15.90:0/3710441451 >> v1:172.21.15.102:6789/0 conn(0x7fde50161e10 legacy=0x7fde501621f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.907+0000 7fde55a00640 1 -- 172.21.15.90:0/3710441451 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde500d1830 con 0x7fde500d0a30 2024-03-31T15:53:46.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde55a00640 1 --2- 172.21.15.90:0/3710441451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde500d0a30 0x7fde500cc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fde500b48b0 tx=0x7fde4801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c7447d39a6be37 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:46.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde4f400640 1 -- 172.21.15.90:0/3710441451 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480a5620 con 0x7fde500d0a30 2024-03-31T15:53:46.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde4f400640 1 -- 172.21.15.90:0/3710441451 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde480a57c0 con 0x7fde500d0a30 2024-03-31T15:53:46.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde4f400640 1 -- 172.21.15.90:0/3710441451 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480a5b90 con 0x7fde500d0a30 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- 172.21.15.90:0/3710441451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde500d0a30 msgr2=0x7fde500cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- 172.21.15.90:0/3710441451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde500d0a30 0x7fde500cc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fde500b48b0 tx=0x7fde4801c960 comp rx=0 tx=0).stop 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- 172.21.15.90:0/3710441451 shutdown_connections 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- 172.21.15.90:0/3710441451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde500d0a30 0x7fde500cc7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- 172.21.15.90:0/3710441451 >> v2:172.21.15.102:3300/0 conn(0x7fde5015e4d0 0x7fde5006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- 172.21.15.90:0/3710441451 >> 172.21.15.90:0/3710441451 conn(0x7fde500d1f80 msgr2=0x7fde500d2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- 172.21.15.90:0/3710441451 shutdown_connections 2024-03-31T15:53:46.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- 172.21.15.90:0/3710441451 wait complete. 2024-03-31T15:53:46.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 Processor -- start 2024-03-31T15:53:46.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- start start 2024-03-31T15:53:46.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 0x7fde50155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 0x7fde50155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde50161e10 0x7fde500c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fde500cd880 con 0x7fde5015e4d0 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde500cd700 con 0x7fde50161e10 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde56400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fde500cda00 con 0x7fde500d0a30 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde4fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 0x7fde50155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.911+0000 7fde55000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 0x7fde50155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde55a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde50161e10 0x7fde500c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde55000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 0x7fde50155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:43316/0 (socket says 172.21.15.90:43316) 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde55000640 1 -- 172.21.15.90:0/2150485255 learned_addr learned my addr 172.21.15.90:0/2150485255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 -- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 msgr2=0x7fde50155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 0x7fde50155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 -- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde50161e10 msgr2=0x7fde500c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:46.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde50161e10 0x7fde500c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:46.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 -- 172.21.15.90:0/2150485255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde50154c50 con 0x7fde5015e4d0 2024-03-31T15:53:46.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4fe00640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 0x7fde50155a10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e870 tx=0x7fde3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c017070 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c00c940 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c00ccf0 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde50154f30 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde50156720 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde500685f0 con 0x7fde5015e4d0 2024-03-31T15:53:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.915+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fde3c01a070 con 0x7fde5015e4d0 2024-03-31T15:53:46.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.919+0000 7fde4cc00640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 0x7fde3006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:46.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.919+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fde3c086ee0 con 0x7fde5015e4d0 2024-03-31T15:53:46.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.919+0000 7fde55000640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 0x7fde3006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:46.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.919+0000 7fde55000640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 0x7fde3006a870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde500d0370 tx=0x7fde40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:46.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:46.923+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fde3c056f90 con 0x7fde5015e4d0 2024-03-31T15:53:47.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.111+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fde5015a210 con 0x7fde5015e4d0 2024-03-31T15:53:47.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:53:47.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:53:47.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.743+0000 7fde4cc00640 1 -- 172.21.15.90:0/2150485255 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fde3c05b290 con 0x7fde5015e4d0 2024-03-31T15:53:47.741 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 2 and data pool 3 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 msgr2=0x7fde3006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 0x7fde3006a870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde500d0370 tx=0x7fde40009040 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 msgr2=0x7fde50155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 0x7fde50155a10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e870 tx=0x7fde3c00ed30 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 shutdown_connections 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde300683b0 0x7fde3006a870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde50161e10 0x7fde500c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde5015e4d0 0x7fde50155a10 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00e870 tx=0x7fde3c00ed30 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 --2- 172.21.15.90:0/2150485255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde500d0a30 0x7fde50155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.755+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 >> 172.21.15.90:0/2150485255 conn(0x7fde500d1f80 msgr2=0x7fde5015aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.759+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 shutdown_connections 2024-03-31T15:53:47.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.759+0000 7fde56400640 1 -- 172.21.15.90:0/2150485255 wait complete. 2024-03-31T15:53:47.784 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-31T15:53:47.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-31T15:53:47.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 Processor -- start 2024-03-31T15:53:47.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 -- start start 2024-03-31T15:53:47.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f128006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f12800cbab0 0x7f128015c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f128015cd60 con 0x7f12800cb630 2024-03-31T15:53:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f128015fdc0 con 0x7f12800cbab0 2024-03-31T15:53:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1287000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f128015d040 con 0x7f128015be10 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1285000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f12800cbab0 0x7f128015c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f127fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f128006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f127f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f12800cb630 0x7f1280160250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56670/0 (socket says 172.21.15.90:56670) 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f127f400640 1 -- 172.21.15.90:0/2397312121 learned_addr learned my addr 172.21.15.90:0/2397312121 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1285000640 1 -- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 msgr2=0x7f128006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1285000640 1 --2- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f128006ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1285000640 1 -- 172.21.15.90:0/2397312121 >> v1:172.21.15.102:6789/0 conn(0x7f12800cb630 legacy=0x7f1280160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f1285000640 1 -- 172.21.15.90:0/2397312121 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128015d320 con 0x7f12800cbab0 2024-03-31T15:53:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.931+0000 7f127ea00640 1 -- 172.21.15.90:0/2397312121 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382884968 0 0) 0x7f128015cd60 con 0x7f12800cb630 2024-03-31T15:53:47.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1285000640 1 --2- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cbab0 0x7f128015c1f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1278009d90 tx=0x7f127801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f5e44fcce8fd726 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:47.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f127ea00640 1 -- 172.21.15.90:0/2397312121 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12780b2070 con 0x7f12800cbab0 2024-03-31T15:53:47.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f127ea00640 1 -- 172.21.15.90:0/2397312121 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12780a57a0 con 0x7f12800cbab0 2024-03-31T15:53:47.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f127ea00640 1 -- 172.21.15.90:0/2397312121 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12780a5b70 con 0x7f12800cbab0 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 -- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cbab0 msgr2=0x7f128015c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 --2- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cbab0 0x7f128015c1f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1278009d90 tx=0x7f127801c8e0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 -- 172.21.15.90:0/2397312121 shutdown_connections 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 --2- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cbab0 0x7f128015c1f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 --2- 172.21.15.90:0/2397312121 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f128006ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 -- 172.21.15.90:0/2397312121 >> 172.21.15.90:0/2397312121 conn(0x7f12800b1880 msgr2=0x7f12800b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 -- 172.21.15.90:0/2397312121 shutdown_connections 2024-03-31T15:53:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.935+0000 7f1287000640 1 -- 172.21.15.90:0/2397312121 wait complete. 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 Processor -- start 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- start start 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 0x7f128014a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f12800ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1280169160 con 0x7f12800cbab0 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12801692e0 con 0x7f12800cb630 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1280169460 con 0x7f128015be10 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33286/0 (socket says 172.21.15.90:33286) 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 -- 172.21.15.90:0/2275126500 learned_addr learned my addr 172.21.15.90:0/2275126500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127fe00640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 0x7f128014a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1285000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f12800ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 -- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 msgr2=0x7f12800ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f12800ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 -- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 msgr2=0x7f128014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 0x7f128014a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 -- 172.21.15.90:0/2275126500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1280156a00 con 0x7f12800cbab0 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127fe00640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 0x7f128014a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:47.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1285000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f12800ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f127f400640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c4a0 tx=0x7f126c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c018070 con 0x7f12800cbab0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126c0074e0 con 0x7f12800cbab0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f128014ab60 con 0x7f12800cbab0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c004a30 con 0x7f12800cbab0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.939+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12801542a0 con 0x7f12800cbab0 2024-03-31T15:53:47.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12800686d0 con 0x7f12800cbab0 2024-03-31T15:53:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f126c01b070 con 0x7f12800cbab0 2024-03-31T15:53:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f1277e00640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 0x7f126006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f126c085d10 con 0x7f12800cbab0 2024-03-31T15:53:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f127fe00640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 0x7f126006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:47.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.943+0000 7f127fe00640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 0x7f126006a870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1270009b10 tx=0x7f1270008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:47.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:47.947+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f126c08b020 con 0x7f12800cbab0 2024-03-31T15:53:48.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.115+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f128006e6a0 con 0x7f12800cbab0 2024-03-31T15:53:48.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1277e00640 1 -- 172.21.15.90:0/2275126500 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f126c056230 con 0x7f12800cbab0 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 msgr2=0x7f126006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 0x7f126006a870 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1270009b10 tx=0x7f1270008040 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 msgr2=0x7f12800ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c4a0 tx=0x7f126c00c960 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 shutdown_connections 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f12600683b0 0x7f126006a870 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f128015be10 0x7f12800ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800cbab0 0x7f12800ce580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 --2- 172.21.15.90:0/2275126500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12800cb630 0x7f128014a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 >> 172.21.15.90:0/2275126500 conn(0x7f12800b1880 msgr2=0x7f1280155130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:48.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 shutdown_connections 2024-03-31T15:53:48.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.735+0000 7f1287000640 1 -- 172.21.15.90:0/2275126500 wait complete. 2024-03-31T15:53:48.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:53:48.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 Processor -- start 2024-03-31T15:53:48.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 -- start start 2024-03-31T15:53:48.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6320144b20 0x7f632006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f63200b47e0 0x7f63200cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f632006ee00 con 0x7f6320072630 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6320144f00 con 0x7f6320144b20 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6327200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f632006f080 con 0x7f63200b47e0 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6320144b20 0x7f632006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6320072630 0x7f63200b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47478/0 (socket says 172.21.15.90:47478) 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325400640 1 -- 172.21.15.90:0/2640397588 learned_addr learned my addr 172.21.15.90:0/2640397588 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6326800640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f63200b47e0 0x7f63200cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325e00640 1 -- 172.21.15.90:0/2640397588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f63200b47e0 msgr2=0x7f63200cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325e00640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f63200b47e0 0x7f63200cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325e00640 1 -- 172.21.15.90:0/2640397588 >> v1:172.21.15.90:6789/0 conn(0x7f6320072630 legacy=0x7f63200b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6325e00640 1 -- 172.21.15.90:0/2640397588 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632006f360 con 0x7f6320144b20 2024-03-31T15:53:48.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.939+0000 7f6324a00640 1 -- 172.21.15.90:0/2640397588 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106495790 0 0) 0x7f6310001120 con 0x7f6320072630 2024-03-31T15:53:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6325e00640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6320144b20 0x7f632006e8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63140097c0 tx=0x7f631401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b813aa254872ad server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:48.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6324a00640 1 -- 172.21.15.90:0/2640397588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63140b2070 con 0x7f6320144b20 2024-03-31T15:53:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6324a00640 1 -- 172.21.15.90:0/2640397588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63140ad640 con 0x7f6320144b20 2024-03-31T15:53:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6324a00640 1 -- 172.21.15.90:0/2640397588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63140ada10 con 0x7f6320144b20 2024-03-31T15:53:48.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 -- 172.21.15.90:0/2640397588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6320144b20 msgr2=0x7f632006e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6320144b20 0x7f632006e8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63140097c0 tx=0x7f631401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 -- 172.21.15.90:0/2640397588 shutdown_connections 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f63200b47e0 0x7f63200cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 --2- 172.21.15.90:0/2640397588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6320144b20 0x7f632006e8c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 -- 172.21.15.90:0/2640397588 >> 172.21.15.90:0/2640397588 conn(0x7f63200cffe0 msgr2=0x7f63200d03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 -- 172.21.15.90:0/2640397588 shutdown_connections 2024-03-31T15:53:48.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.943+0000 7f6327200640 1 -- 172.21.15.90:0/2640397588 wait complete. 2024-03-31T15:53:48.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 Processor -- start 2024-03-31T15:53:48.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- start start 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 0x7f6320154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 0x7f6320140a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f63200ce820 con 0x7f63200b47e0 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f63200ce6a0 con 0x7f6320072630 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f63200ce9a0 con 0x7f6320144b20 2024-03-31T15:53:48.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33302/0 (socket says 172.21.15.90:33302) 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 -- 172.21.15.90:0/3634654637 learned_addr learned my addr 172.21.15.90:0/3634654637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325e00640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 0x7f6320154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6326800640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 0x7f6320140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 -- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 msgr2=0x7f6320140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 0x7f6320140a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 -- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 msgr2=0x7f6320154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 0x7f6320154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 -- 172.21.15.90:0/3634654637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6320153390 con 0x7f63200b47e0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6326800640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 0x7f6320140a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325e00640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 0x7f6320154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6325400640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f631000c780 tx=0x7f631000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310018070 con 0x7f63200b47e0 2024-03-31T15:53:48.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63100072f0 con 0x7f63200b47e0 2024-03-31T15:53:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6320153670 con 0x7f63200b47e0 2024-03-31T15:53:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63100047f0 con 0x7f63200b47e0 2024-03-31T15:53:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.947+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6320153c30 con 0x7f63200b47e0 2024-03-31T15:53:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.951+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6320068d00 con 0x7f63200b47e0 2024-03-31T15:53:48.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.951+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f631001b070 con 0x7f63200b47e0 2024-03-31T15:53:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.951+0000 7f631a000640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 0x7f630406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:48.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.951+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f6310085ea0 con 0x7f63200b47e0 2024-03-31T15:53:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.951+0000 7f6325e00640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 0x7f630406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:48.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.955+0000 7f6325e00640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 0x7f630406a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63140af000 tx=0x7f6314008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:48.959+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f632006ee00 con 0x7f63200b47e0 2024-03-31T15:53:49.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:49.115+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f632015ba50 con 0x7f63200b47e0 2024-03-31T15:53:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.759+0000 7f631a000640 1 -- 172.21.15.90:0/3634654637 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f6310055c40 con 0x7f63200b47e0 2024-03-31T15:53:50.753 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' created 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 msgr2=0x7f630406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 0x7f630406a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63140af000 tx=0x7f6314008040 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 msgr2=0x7f632013bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f631000c780 tx=0x7f631000cc40 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 shutdown_connections 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6304068380 0x7f630406a840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6320144b20 0x7f6320140a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f63200b47e0 0x7f632013bbd0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f631000c780 tx=0x7f631000cc40 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 --2- 172.21.15.90:0/3634654637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6320072630 0x7f6320154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 >> 172.21.15.90:0/3634654637 conn(0x7f63200cffe0 msgr2=0x7f63200cf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 shutdown_connections 2024-03-31T15:53:50.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.763+0000 7f6327200640 1 -- 172.21.15.90:0/3634654637 wait complete. 2024-03-31T15:53:50.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-31T15:53:50.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.935+0000 7f5f3a800640 1 Processor -- start 2024-03-31T15:53:50.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.935+0000 7f5f3a800640 1 -- start start 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f3a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 0x7f5f340cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f3a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5f3406e170 0x7f5f34162460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f3a800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f340b4260 con 0x7f5f34162000 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f3a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5f341629a0 con 0x7f5f3406e170 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f3a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3406d5d0 con 0x7f5f3415e3b0 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f2be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5f34162000 0x7f5f3406da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47482/0 (socket says 172.21.15.90:47482) 2024-03-31T15:53:50.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f2be00640 1 -- 172.21.15.90:0/3641102588 learned_addr learned my addr 172.21.15.90:0/3641102588 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33400640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 0x7f5f340cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3641102588 >> v2:172.21.15.102:3300/0 conn(0x7f5f3406e170 0x7f5f34162460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f32a00640 1 -- 172.21.15.90:0/3641102588 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3051321886 0 0) 0x7f5f340b4260 con 0x7f5f34162000 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f32a00640 1 -- 172.21.15.90:0/3641102588 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f18003530 con 0x7f5f34162000 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33e00640 1 -- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 msgr2=0x7f5f340cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 0x7f5f340cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33e00640 1 -- 172.21.15.90:0/3641102588 >> v1:172.21.15.90:6789/0 conn(0x7f5f34162000 legacy=0x7f5f3406da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33e00640 1 -- 172.21.15.90:0/3641102588 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340b4540 con 0x7f5f3406e170 2024-03-31T15:53:50.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.939+0000 7f5f33400640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 0x7f5f340cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:50.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3406e170 0x7f5f34162460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009d90 tx=0x7f5f2c01c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb338409597e6ef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:50.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f32a00640 1 -- 172.21.15.90:0/3641102588 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0b2070 con 0x7f5f3406e170 2024-03-31T15:53:50.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f32a00640 1 -- 172.21.15.90:0/3641102588 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c0a5a00 con 0x7f5f3406e170 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f32a00640 1 -- 172.21.15.90:0/3641102588 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0a5dd0 con 0x7f5f3406e170 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3406e170 msgr2=0x7f5f34162460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3406e170 0x7f5f34162460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009d90 tx=0x7f5f2c01c940 comp rx=0 tx=0).stop 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- 172.21.15.90:0/3641102588 shutdown_connections 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3406e170 0x7f5f34162460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3641102588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3415e3b0 0x7f5f340cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- 172.21.15.90:0/3641102588 >> 172.21.15.90:0/3641102588 conn(0x7f5f3409aa40 msgr2=0x7f5f3409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- 172.21.15.90:0/3641102588 shutdown_connections 2024-03-31T15:53:50.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- 172.21.15.90:0/3641102588 wait complete. 2024-03-31T15:53:50.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 Processor -- start 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.943+0000 7f5f3a800640 1 -- start start 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3406e170 0x7f5f34155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3415e3b0 0x7f5f340c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3415e910 con 0x7f5f34162000 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3415e790 con 0x7f5f3415e3b0 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340b3d40 con 0x7f5f3406e170 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33310/0 (socket says 172.21.15.90:33310) 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 -- 172.21.15.90:0/3901611637 learned_addr learned my addr 172.21.15.90:0/3901611637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33400640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3406e170 0x7f5f34155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f2be00640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3415e3b0 0x7f5f340c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 -- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3406e170 msgr2=0x7f5f34155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3406e170 0x7f5f34155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 -- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3415e3b0 msgr2=0x7f5f340c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3415e3b0 0x7f5f340c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 -- 172.21.15.90:0/3901611637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3415f7a0 con 0x7f5f34162000 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f33e00640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c01aff0 tx=0x7f5f2c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0b2050 con 0x7f5f34162000 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c0be070 con 0x7f5f34162000 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f340c63e0 con 0x7f5f34162000 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0ada70 con 0x7f5f34162000 2024-03-31T15:53:50.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f340c69a0 con 0x7f5f34162000 2024-03-31T15:53:50.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.947+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f340686d0 con 0x7f5f34162000 2024-03-31T15:53:50.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.951+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5f2c0adc10 con 0x7f5f34162000 2024-03-31T15:53:50.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.951+0000 7f5f2b400640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 0x7f5f1006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:50.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.951+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f5f2c122830 con 0x7f5f34162000 2024-03-31T15:53:50.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.951+0000 7f5f33400640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 0x7f5f1006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:50.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.951+0000 7f5f33400640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 0x7f5f1006a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f341fa7b0 tx=0x7f5f24009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:50.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:50.959+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5f2c016560 con 0x7f5f34162000 2024-03-31T15:53:51.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.115+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f5f341579b0 con 0x7f5f34162000 2024-03-31T15:53:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.759+0000 7f5f2b400640 1 -- 172.21.15.90:0/3901611637 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f5f2c0f24f0 con 0x7f5f34162000 2024-03-31T15:53:51.755 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 4 allow_ec_overwrites to true 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 msgr2=0x7f5f1006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 0x7f5f1006a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5f341fa7b0 tx=0x7f5f24009190 comp rx=0 tx=0).stop 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 msgr2=0x7f5f34158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c01aff0 tx=0x7f5f2c0196e0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 shutdown_connections 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5f100683b0 0x7f5f1006a870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f34162000 0x7f5f34158010 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f3415e3b0 0x7f5f340c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 --2- 172.21.15.90:0/3901611637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5f3406e170 0x7f5f34155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 >> 172.21.15.90:0/3901611637 conn(0x7f5f3409aa40 msgr2=0x7f5f340cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:51.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 shutdown_connections 2024-03-31T15:53:51.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.763+0000 7f5f3a800640 1 -- 172.21.15.90:0/3901611637 wait complete. 2024-03-31T15:53:51.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-31T15:53:51.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 Processor -- start 2024-03-31T15:53:51.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 -- start start 2024-03-31T15:53:51.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:51.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c0c0cefc0 con 0x7f4c0c0cd640 2024-03-31T15:53:51.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c0c0cf2a0 con 0x7f4c0c0d2b20 2024-03-31T15:53:51.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.947+0000 7f4c11a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4c0c0aed70 con 0x7f4c0c15c320 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c0c0cd640 0x7f4c0c0af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:47492/0 (socket says 172.21.15.90:47492) 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33326/0 (socket says 172.21.15.90:33326) 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0b400640 1 -- 172.21.15.90:0/1788933675 learned_addr learned my addr 172.21.15.90:0/1788933675 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 -- 172.21.15.90:0/1788933675 >> v1:172.21.15.102:6789/0 conn(0x7f4c0c0d2b20 legacy=0x7f4c0c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 -- 172.21.15.90:0/1788933675 >> v1:172.21.15.90:6789/0 conn(0x7f4c0c0cd640 legacy=0x7f4c0c0af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 -- 172.21.15.90:0/1788933675 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0c0cf580 con 0x7f4c0c15c320 2024-03-31T15:53:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0aa00640 1 --2- 172.21.15.90:0/1788933675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097c0 tx=0x7f4bfc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5dc673dba8d0799 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0a000640 1 -- 172.21.15.90:0/1788933675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0b0070 con 0x7f4c0c15c320 2024-03-31T15:53:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0a000640 1 -- 172.21.15.90:0/1788933675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c0c0d0f20 con 0x7f4c0c15c320 2024-03-31T15:53:51.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.951+0000 7f4c0a000640 1 -- 172.21.15.90:0/1788933675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0ab820 con 0x7f4c0c15c320 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- 172.21.15.90:0/1788933675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 msgr2=0x7f4c0c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1788933675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0097c0 tx=0x7f4bfc01c410 comp rx=0 tx=0).stop 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- 172.21.15.90:0/1788933675 shutdown_connections 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1788933675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c0cc9f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- 172.21.15.90:0/1788933675 >> 172.21.15.90:0/1788933675 conn(0x7f4c0c0d1f20 msgr2=0x7f4c0c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- 172.21.15.90:0/1788933675 shutdown_connections 2024-03-31T15:53:51.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- 172.21.15.90:0/1788933675 wait complete. 2024-03-31T15:53:51.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 Processor -- start 2024-03-31T15:53:51.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- start start 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 0x7f4c0c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c0c0d2b20 0x7f4c0c153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c0b04f0 con 0x7f4c0c15c320 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4c0c0b0370 con 0x7f4c0c0cd640 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c11a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4c0c169ed0 con 0x7f4c0c0d2b20 2024-03-31T15:53:51.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:33334/0 (socket says 172.21.15.90:33334) 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 -- 172.21.15.90:0/1228921823 learned_addr learned my addr 172.21.15.90:0/1228921823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0aa00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 0x7f4c0c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c03e00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c0c0d2b20 0x7f4c0c153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 -- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c0c0d2b20 msgr2=0x7f4c0c153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c0c0d2b20 0x7f4c0c153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 -- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 msgr2=0x7f4c0c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 0x7f4c0c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 -- 172.21.15.90:0/1228921823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0c0c76d0 con 0x7f4c0c15c320 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0aa00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 0x7f4c0c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.955+0000 7f4c0b400640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400cad0 tx=0x7f4c0400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04017070 con 0x7f4c0c15c320 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c0400ac30 con 0x7f4c0c15c320 2024-03-31T15:53:51.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04012400 con 0x7f4c0c15c320 2024-03-31T15:53:51.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0c0c79b0 con 0x7f4c0c15c320 2024-03-31T15:53:51.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c0c15f620 con 0x7f4c0c15c320 2024-03-31T15:53:51.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0c0685f0 con 0x7f4c0c15c320 2024-03-31T15:53:51.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4c0401a070 con 0x7f4c0c15c320 2024-03-31T15:53:51.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 0x7f4be806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:51.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f4c040868f0 con 0x7f4c0c15c320 2024-03-31T15:53:51.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.959+0000 7f4c0aa00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 0x7f4be806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:51.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.963+0000 7f4c0aa00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 0x7f4be806a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0ad000 tx=0x7f4bfc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:51.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:51.963+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4c04056530 con 0x7f4c0c15c320 2024-03-31T15:53:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.159+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f4c0c0c92a0 con 0x7f4c0c15c320 2024-03-31T15:53:52.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c02a00640 1 -- 172.21.15.90:0/1228921823 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f4c0405a830 con 0x7f4c0c15c320 2024-03-31T15:53:52.784 INFO:teuthology.orchestra.run.smithi090.stderr:added data pool 4 to fsmap 2024-03-31T15:53:52.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 msgr2=0x7f4be806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 0x7f4be806a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0ad000 tx=0x7f4bfc0a6000 comp rx=0 tx=0).stop 2024-03-31T15:53:52.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 msgr2=0x7f4c0c1542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400cad0 tx=0x7f4c0400cf90 comp rx=0 tx=0).stop 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 shutdown_connections 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4be8068380 0x7f4be806a840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c0c15c320 0x7f4c0c1542d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c0c0d2b20 0x7f4c0c153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 --2- 172.21.15.90:0/1228921823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c0c0cd640 0x7f4c0c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 >> 172.21.15.90:0/1228921823 conn(0x7f4c0c0d1f20 msgr2=0x7f4c0c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 shutdown_connections 2024-03-31T15:53:52.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.791+0000 7f4c11a00640 1 -- 172.21.15.90:0/1228921823 wait complete. 2024-03-31T15:53:52.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:53:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.955+0000 7f37d8e00640 1 Processor -- start 2024-03-31T15:53:52.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.955+0000 7f37d8e00640 1 -- start start 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d8e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37d415fd00 0x7f37d40d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37d40cca30 0x7f37d40cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d8e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d4160350 con 0x7f37d415c120 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d8e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f37d4160bf0 con 0x7f37d415fd00 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d8e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f37d40cd3a0 con 0x7f37d40cca30 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37d415fd00 0x7f37d40d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f37d415fd00 0x7f37d40d2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43042/0 (socket says 172.21.15.90:43042) 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 -- 172.21.15.90:0/1746581523 learned_addr learned my addr 172.21.15.90:0/1746581523 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37d2800640 1 --2- 172.21.15.90:0/1746581523 >> v2:172.21.15.102:3300/0 conn(0x7f37d40cca30 0x7f37d40cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 -- 172.21.15.90:0/1746581523 >> v2:172.21.15.102:3300/0 conn(0x7f37d40cca30 msgr2=0x7f37d40cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 --2- 172.21.15.90:0/1746581523 >> v2:172.21.15.102:3300/0 conn(0x7f37d40cca30 0x7f37d40cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 -- 172.21.15.90:0/1746581523 >> v1:172.21.15.90:6789/0 conn(0x7f37d415c120 legacy=0x7f37d40b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.959+0000 7f37cbe00640 1 -- 172.21.15.90:0/1746581523 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d4160630 con 0x7f37d415fd00 2024-03-31T15:53:52.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37cbe00640 1 --2- 172.21.15.90:0/1746581523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d415fd00 0x7f37d40d2680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37d40d3000 tx=0x7f37bc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e7d699cbd59329 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d1400640 1 -- 172.21.15.90:0/1746581523 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0b1070 con 0x7f37d415fd00 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d1400640 1 -- 172.21.15.90:0/1746581523 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc0ac600 con 0x7f37d415fd00 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d1400640 1 -- 172.21.15.90:0/1746581523 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0ac9d0 con 0x7f37d415fd00 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- 172.21.15.90:0/1746581523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d415fd00 msgr2=0x7f37d40d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 --2- 172.21.15.90:0/1746581523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d415fd00 0x7f37d40d2680 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37d40d3000 tx=0x7f37bc01c490 comp rx=0 tx=0).stop 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- 172.21.15.90:0/1746581523 shutdown_connections 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 --2- 172.21.15.90:0/1746581523 >> v2:172.21.15.102:3300/0 conn(0x7f37d40cca30 0x7f37d40cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 --2- 172.21.15.90:0/1746581523 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d415fd00 0x7f37d40d2680 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- 172.21.15.90:0/1746581523 >> 172.21.15.90:0/1746581523 conn(0x7f37d409aa40 msgr2=0x7f37d409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- 172.21.15.90:0/1746581523 shutdown_connections 2024-03-31T15:53:52.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- 172.21.15.90:0/1746581523 wait complete. 2024-03-31T15:53:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 Processor -- start 2024-03-31T15:53:52.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.963+0000 7f37d8e00640 1 -- start start 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37d415c120 0x7f37d40c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37d415fd00 0x7f37d4154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37d40b2f70 con 0x7f37d40cca30 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f37d40b2df0 con 0x7f37d415c120 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f37d4169040 con 0x7f37d415fd00 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43052/0 (socket says 172.21.15.90:43052) 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 -- 172.21.15.90:0/2846591341 learned_addr learned my addr 172.21.15.90:0/2846591341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37cbe00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37d415c120 0x7f37d40c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d2800640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37d415fd00 0x7f37d4154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 -- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37d415fd00 msgr2=0x7f37d4154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37d415fd00 0x7f37d4154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 -- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37d415c120 msgr2=0x7f37d40c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37d415c120 0x7f37d40c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 -- 172.21.15.90:0/2846591341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d40c7e60 con 0x7f37d40cca30 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d1e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37c400c370 tx=0x7f37c400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c40052d0 con 0x7f37d40cca30 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c4005470 con 0x7f37d40cca30 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d40c8140 con 0x7f37d40cca30 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4007260 con 0x7f37d40cca30 2024-03-31T15:53:52.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d40c8700 con 0x7f37d40cca30 2024-03-31T15:53:52.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d40686d0 con 0x7f37d40cca30 2024-03-31T15:53:52.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.967+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f37c4007400 con 0x7f37d40cca30 2024-03-31T15:53:52.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.971+0000 7f37ca000640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 0x7f37b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:52.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.971+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f37c4086100 con 0x7f37d40cca30 2024-03-31T15:53:52.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.971+0000 7f37cbe00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 0x7f37b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:52.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.971+0000 7f37cbe00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 0x7f37b006a840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0ae000 tx=0x7f37bc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:52.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:52.975+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f37c4055d20 con 0x7f37d40cca30 2024-03-31T15:53:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.163+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37d406fd60 con 0x7f37d40cca30 2024-03-31T15:53:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.163+0000 7f37ca000640 1 -- 172.21.15.90:0/2846591341 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7f37c405c020 con 0x7f37d40cca30 2024-03-31T15:53:53.157 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:53.158 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:53:52.163102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:53:53.158 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-03-31T15:53:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 msgr2=0x7f37b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 0x7f37b006a840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0ae000 tx=0x7f37bc0a7000 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 msgr2=0x7f37d41f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37c400c370 tx=0x7f37c400c830 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 shutdown_connections 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37b0068380 0x7f37b006a840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37d415fd00 0x7f37d4154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37d415c120 0x7f37d40c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 --2- 172.21.15.90:0/2846591341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37d40cca30 0x7f37d41f73e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 >> 172.21.15.90:0/2846591341 conn(0x7f37d409aa40 msgr2=0x7f37d409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 shutdown_connections 2024-03-31T15:53:53.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.167+0000 7f37d8e00640 1 -- 172.21.15.90:0/2846591341 wait complete. 2024-03-31T15:53:53.178 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:53.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 Processor -- start 2024-03-31T15:53:53.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 -- start start 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f452c0cc9d0 0x7f452c142030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452c0b34f0 con 0x7f452c06b820 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f452c0b37d0 con 0x7f452c0ccdb0 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4533c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f452c0b3ab0 con 0x7f452c0cc9d0 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f452c0cc9d0 0x7f452c142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f452c06b820 0x7f452c0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57792/0 (socket says 172.21.15.90:57792) 2024-03-31T15:53:53.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531c00640 1 -- 172.21.15.90:0/1845357930 learned_addr learned my addr 172.21.15.90:0/1845357930 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:53.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531200640 1 -- 172.21.15.90:0/1845357930 >> v1:172.21.15.102:6789/0 conn(0x7f452c0ccdb0 legacy=0x7f452c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531200640 1 -- 172.21.15.90:0/1845357930 >> v1:172.21.15.90:6789/0 conn(0x7f452c06b820 legacy=0x7f452c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f452b400640 1 -- 172.21.15.90:0/1845357930 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744290848 0 0) 0x7f452c0b34f0 con 0x7f452c06b820 2024-03-31T15:53:53.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.323+0000 7f4531200640 1 -- 172.21.15.90:0/1845357930 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c141830 con 0x7f452c0cc9d0 2024-03-31T15:53:53.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4531200640 1 --2- 172.21.15.90:0/1845357930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0cc9d0 0x7f452c142030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45200097c0 tx=0x7f452001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168c9b76f8332e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f452b400640 1 -- 172.21.15.90:0/1845357930 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45200b0070 con 0x7f452c0cc9d0 2024-03-31T15:53:53.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f452b400640 1 -- 172.21.15.90:0/1845357930 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45200ab600 con 0x7f452c0cc9d0 2024-03-31T15:53:53.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f452b400640 1 -- 172.21.15.90:0/1845357930 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45200ab9d0 con 0x7f452c0cc9d0 2024-03-31T15:53:53.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- 172.21.15.90:0/1845357930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0cc9d0 msgr2=0x7f452c142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 --2- 172.21.15.90:0/1845357930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0cc9d0 0x7f452c142030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f45200097c0 tx=0x7f452001c410 comp rx=0 tx=0).stop 2024-03-31T15:53:53.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- 172.21.15.90:0/1845357930 shutdown_connections 2024-03-31T15:53:53.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 --2- 172.21.15.90:0/1845357930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0cc9d0 0x7f452c142030 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- 172.21.15.90:0/1845357930 >> 172.21.15.90:0/1845357930 conn(0x7f452c0b1c80 msgr2=0x7f452c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:53.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- 172.21.15.90:0/1845357930 shutdown_connections 2024-03-31T15:53:53.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- 172.21.15.90:0/1845357930 wait complete. 2024-03-31T15:53:53.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 Processor -- start 2024-03-31T15:53:53.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.327+0000 7f4533c00640 1 -- start start 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452c06b820 0x7f452c0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f452c0cc9d0 0x7f452c153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f452c16a7d0 con 0x7f452c0ccdb0 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f452c0b3290 con 0x7f452c06b820 2024-03-31T15:53:53.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f452c16a950 con 0x7f452c0cc9d0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43076/0 (socket says 172.21.15.90:43076) 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 -- 172.21.15.90:0/1565897062 learned_addr learned my addr 172.21.15.90:0/1565897062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531200640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452c06b820 0x7f452c0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f452be00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f452c0cc9d0 0x7f452c153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 -- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f452c0cc9d0 msgr2=0x7f452c153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f452c0cc9d0 0x7f452c153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 -- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452c06b820 msgr2=0x7f452c0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452c06b820 0x7f452c0cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 -- 172.21.15.90:0/1565897062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c13dce0 con 0x7f452c0ccdb0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4531c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f452400c700 tx=0x7f452400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524018070 con 0x7f452c0ccdb0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f452400ce10 con 0x7f452c0ccdb0 2024-03-31T15:53:53.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452400a880 con 0x7f452c0ccdb0 2024-03-31T15:53:53.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c13da00 con 0x7f452c0ccdb0 2024-03-31T15:53:53.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c071770 con 0x7f452c0ccdb0 2024-03-31T15:53:53.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.331+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c0686d0 con 0x7f452c0ccdb0 2024-03-31T15:53:53.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.335+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f452400aa20 con 0x7f452c0ccdb0 2024-03-31T15:53:53.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.335+0000 7f4528c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.335+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4524086210 con 0x7f452c0ccdb0 2024-03-31T15:53:53.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.335+0000 7f4531200640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.335+0000 7f4531200640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 0x7f450c06a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45200ad000 tx=0x7f45200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.339+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f452c13dce0 con 0x7f452c0ccdb0 2024-03-31T15:53:53.384 INFO:tasks.ceph.mds.a.smithi090.stderr:2024-03-31T15:53:53.391+0000 7fc431c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:53:53.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.499+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f452c167d90 con 0x7f452c0ccdb0 2024-03-31T15:53:53.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.499+0000 7f4528c00640 1 -- 172.21.15.90:0/1565897062 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f4524055e20 con 0x7f452c0ccdb0 2024-03-31T15:53:53.494 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:53.494 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 msgr2=0x7f450c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 0x7f450c06a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45200ad000 tx=0x7f45200a6000 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 msgr2=0x7f452c13d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f452400c700 tx=0x7f452400cbc0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 shutdown_connections 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f450c068380 0x7f450c06a840 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f452c0ccdb0 0x7f452c13d2d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f452c0cc9d0 0x7f452c153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 --2- 172.21.15.90:0/1565897062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452c06b820 0x7f452c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 >> 172.21.15.90:0/1565897062 conn(0x7f452c0b1c80 msgr2=0x7f452c0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 shutdown_connections 2024-03-31T15:53:53.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.503+0000 7f4533c00640 1 -- 172.21.15.90:0/1565897062 wait complete. 2024-03-31T15:53:53.515 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:53.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 Processor -- start 2024-03-31T15:53:53.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 -- start start 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38cc15f390 0x7f38cc0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38cc067330 0x7f38cc072d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38cc15fb10 con 0x7f38cc067790 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f38cc160a00 con 0x7f38cc067330 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f38cc15fdf0 con 0x7f38cc15f390 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38cc067330 0x7f38cc072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38cc067330 0x7f38cc072d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43092/0 (socket says 172.21.15.90:43092) 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 -- 172.21.15.90:0/806008480 learned_addr learned my addr 172.21.15.90:0/806008480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d2400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f38cc067790 0x7f38cc0d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57806/0 (socket says 172.21.15.90:57806) 2024-03-31T15:53:53.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1a00640 1 --2- 172.21.15.90:0/806008480 >> v2:172.21.15.102:3300/0 conn(0x7f38cc15f390 0x7f38cc0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 -- 172.21.15.90:0/806008480 >> v2:172.21.15.102:3300/0 conn(0x7f38cc15f390 msgr2=0x7f38cc0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:53.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 --2- 172.21.15.90:0/806008480 >> v2:172.21.15.102:3300/0 conn(0x7f38cc15f390 0x7f38cc0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 -- 172.21.15.90:0/806008480 >> v1:172.21.15.90:6789/0 conn(0x7f38cc067790 legacy=0x7f38cc0d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38d1000640 1 -- 172.21.15.90:0/806008480 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc1600d0 con 0x7f38cc067330 2024-03-31T15:53:53.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.659+0000 7f38cbe00640 1 -- 172.21.15.90:0/806008480 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337681448 0 0) 0x7f38cc15fb10 con 0x7f38cc067790 2024-03-31T15:53:53.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.663+0000 7f38d1000640 1 --2- 172.21.15.90:0/806008480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc072d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009870 tx=0x7f38b801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a432c15f7273d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.663+0000 7f38cbe00640 1 -- 172.21.15.90:0/806008480 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80b0070 con 0x7f38cc067330 2024-03-31T15:53:53.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.663+0000 7f38cbe00640 1 -- 172.21.15.90:0/806008480 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b80ab600 con 0x7f38cc067330 2024-03-31T15:53:53.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.663+0000 7f38cbe00640 1 -- 172.21.15.90:0/806008480 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b80ab9d0 con 0x7f38cc067330 2024-03-31T15:53:53.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- 172.21.15.90:0/806008480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 msgr2=0x7f38cc072d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- 172.21.15.90:0/806008480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc072d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38b8009870 tx=0x7f38b801c460 comp rx=0 tx=0).stop 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- 172.21.15.90:0/806008480 shutdown_connections 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- 172.21.15.90:0/806008480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc072d30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- 172.21.15.90:0/806008480 >> v2:172.21.15.102:3300/0 conn(0x7f38cc15f390 0x7f38cc0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- 172.21.15.90:0/806008480 >> 172.21.15.90:0/806008480 conn(0x7f38cc0b5130 msgr2=0x7f38cc0b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- 172.21.15.90:0/806008480 shutdown_connections 2024-03-31T15:53:53.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- 172.21.15.90:0/806008480 wait complete. 2024-03-31T15:53:53.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 Processor -- start 2024-03-31T15:53:53.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- start start 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc067790 0x7f38cc1fb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38cc15f390 0x7f38cc149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc15f8f0 con 0x7f38cc067330 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc15f770 con 0x7f38cc067790 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d2e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc168e70 con 0x7f38cc15f390 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43106/0 (socket says 172.21.15.90:43106) 2024-03-31T15:53:53.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 -- 172.21.15.90:0/3626449337 learned_addr learned my addr 172.21.15.90:0/3626449337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 -- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38cc15f390 msgr2=0x7f38cc149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38cc15f390 0x7f38cc149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 -- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc067790 msgr2=0x7f38cc1fb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc067790 0x7f38cc1fb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 -- 172.21.15.90:0/3626449337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc14a430 con 0x7f38cc067330 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38d1a00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38c000e870 tx=0x7f38c000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.667+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0017070 con 0x7f38cc067330 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c000c940 con 0x7f38cc067330 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc06db80 con 0x7f38cc067330 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c000ccf0 con 0x7f38cc067330 2024-03-31T15:53:53.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc06e140 con 0x7f38cc067330 2024-03-31T15:53:53.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38cc068d00 con 0x7f38cc067330 2024-03-31T15:53:53.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38c001a070 con 0x7f38cc067330 2024-03-31T15:53:53.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38c9600640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:53.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f38c0086d40 con 0x7f38cc067330 2024-03-31T15:53:53.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.671+0000 7f38d1000640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:53.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.675+0000 7f38d1000640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 0x7f38ac06a840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0d1400 tx=0x7f38b8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:53.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.683+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f38c0056ea0 con 0x7f38cc067330 2024-03-31T15:53:53.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.839+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38cc0724c0 con 0x7f38cc067330 2024-03-31T15:53:53.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38c9600640 1 -- 172.21.15.90:0/3626449337 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f38c00567a0 con 0x7f38cc067330 2024-03-31T15:53:53.836 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:53.836 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 msgr2=0x7f38ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 0x7f38ac06a840 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38cc0d1400 tx=0x7f38b8004740 comp rx=0 tx=0).stop 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 msgr2=0x7f38cc15db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38c000e870 tx=0x7f38c000ec70 comp rx=0 tx=0).stop 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 shutdown_connections 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38ac068380 0x7f38ac06a840 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38cc15f390 0x7f38cc149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38cc067790 0x7f38cc1fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 --2- 172.21.15.90:0/3626449337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38cc067330 0x7f38cc15db30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:53.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 >> 172.21.15.90:0/3626449337 conn(0x7f38cc0b5130 msgr2=0x7f38cc14af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:53.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 shutdown_connections 2024-03-31T15:53:53.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:53.843+0000 7f38d2e00640 1 -- 172.21.15.90:0/3626449337 wait complete. 2024-03-31T15:53:53.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:53:54.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.027+0000 7fad67e00640 1 Processor -- start 2024-03-31T15:53:54.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.027+0000 7fad67e00640 1 -- start start 2024-03-31T15:53:54.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad67e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad6015fd00 0x7fad600d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad67e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad67e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad60160350 con 0x7fad6015c120 2024-03-31T15:53:54.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad67e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fad60160bf0 con 0x7fad6015fd00 2024-03-31T15:53:54.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad67e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fad60160630 con 0x7fad600cca30 2024-03-31T15:53:54.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad64a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad6015fd00 0x7fad600d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:40332/0 (socket says 172.21.15.90:40332) 2024-03-31T15:53:54.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 -- 172.21.15.90:0/2370854181 learned_addr learned my addr 172.21.15.90:0/2370854181 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:54.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65400640 1 --1- 172.21.15.90:0/2370854181 >> v1:172.21.15.102:6789/0 conn(0x7fad6015c120 0x7fad600b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:35260/0 (socket says 172.21.15.90:35260) 2024-03-31T15:53:54.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 -- 172.21.15.90:0/2370854181 >> v2:172.21.15.102:3300/0 conn(0x7fad6015fd00 msgr2=0x7fad600d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:54.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 --2- 172.21.15.90:0/2370854181 >> v2:172.21.15.102:3300/0 conn(0x7fad6015fd00 0x7fad600d2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 -- 172.21.15.90:0/2370854181 >> v1:172.21.15.102:6789/0 conn(0x7fad6015c120 legacy=0x7fad600b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.031+0000 7fad65e00640 1 -- 172.21.15.90:0/2370854181 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad60160910 con 0x7fad600cca30 2024-03-31T15:53:54.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad65e00640 1 --2- 172.21.15.90:0/2370854181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad58009e50 tx=0x7fad5801c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb3567ad11abf4e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad5fe00640 1 -- 172.21.15.90:0/2370854181 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad580a5770 con 0x7fad600cca30 2024-03-31T15:53:54.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad5fe00640 1 -- 172.21.15.90:0/2370854181 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad580a5910 con 0x7fad600cca30 2024-03-31T15:53:54.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad5fe00640 1 -- 172.21.15.90:0/2370854181 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad580a5ce0 con 0x7fad600cca30 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- 172.21.15.90:0/2370854181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 msgr2=0x7fad600ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- 172.21.15.90:0/2370854181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad58009e50 tx=0x7fad5801c9a0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- 172.21.15.90:0/2370854181 shutdown_connections 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- 172.21.15.90:0/2370854181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad600cca30 0x7fad600ccea0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- 172.21.15.90:0/2370854181 >> v2:172.21.15.102:3300/0 conn(0x7fad6015fd00 0x7fad600d2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- 172.21.15.90:0/2370854181 >> 172.21.15.90:0/2370854181 conn(0x7fad6009aa40 msgr2=0x7fad6009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:54.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- 172.21.15.90:0/2370854181 shutdown_connections 2024-03-31T15:53:54.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- 172.21.15.90:0/2370854181 wait complete. 2024-03-31T15:53:54.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 Processor -- start 2024-03-31T15:53:54.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- start start 2024-03-31T15:53:54.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600cca30 0x7fad600c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad6015c120 0x7fad600c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad601691c0 con 0x7fad6015fd00 2024-03-31T15:53:54.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fad60169040 con 0x7fad600cca30 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad67e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fad60169340 con 0x7fad6015c120 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad65e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad65e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43122/0 (socket says 172.21.15.90:43122) 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad65e00640 1 -- 172.21.15.90:0/2099952957 learned_addr learned my addr 172.21.15.90:0/2099952957 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad65400640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600cca30 0x7fad600c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.035+0000 7fad64a00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad6015c120 0x7fad600c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 -- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad6015c120 msgr2=0x7fad600c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad6015c120 0x7fad600c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 -- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600cca30 msgr2=0x7fad600c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600cca30 0x7fad600c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 -- 172.21.15.90:0/2099952957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6015f660 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad65e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad580af040 tx=0x7fad58019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad580bc020 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad580b8670 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6015edc0 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58009510 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6015f380 con 0x7fad6015fd00 2024-03-31T15:53:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad600686d0 con 0x7fad6015fd00 2024-03-31T15:53:54.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fad580bf020 con 0x7fad6015fd00 2024-03-31T15:53:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.039+0000 7fad5d600640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 0x7fad4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.043+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fad581217b0 con 0x7fad6015fd00 2024-03-31T15:53:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.043+0000 7fad65400640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 0x7fad4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.043+0000 7fad65400640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 0x7fad4006a870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad54000f70 tx=0x7fad54007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.047+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fad580c6070 con 0x7fad6015fd00 2024-03-31T15:53:54.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.235+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad60074270 con 0x7fad6015fd00 2024-03-31T15:53:54.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.235+0000 7fad5d600640 1 -- 172.21.15.90:0/2099952957 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5177 (secure 0 0 0) 0x7fad6015f660 con 0x7fad6015fd00 2024-03-31T15:53:54.231 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:54.231 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:53:52.163102+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:53:54.231 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 7 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 msgr2=0x7fad4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 0x7fad4006a870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad54000f70 tx=0x7fad54007e90 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 msgr2=0x7fad6015ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fad580af040 tx=0x7fad58019740 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 shutdown_connections 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad400683b0 0x7fad4006a870 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad6015fd00 0x7fad6015ddd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad6015c120 0x7fad600c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 --2- 172.21.15.90:0/2099952957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600cca30 0x7fad600c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 >> 172.21.15.90:0/2099952957 conn(0x7fad6009aa40 msgr2=0x7fad6015b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:54.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 shutdown_connections 2024-03-31T15:53:54.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.239+0000 7fad67e00640 1 -- 172.21.15.90:0/2099952957 wait complete. 2024-03-31T15:53:54.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:54.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 Processor -- start 2024-03-31T15:53:54.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 -- start start 2024-03-31T15:53:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 0x7ff8d415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 0x7ff8d40ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d4160350 con 0x7ff8d40d2680 2024-03-31T15:53:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d4160bf0 con 0x7ff8d40cca30 2024-03-31T15:53:54.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d9800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d4160630 con 0x7ff8d415c120 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 0x7ff8d40ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 0x7ff8d415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff8d40d2680 0x7ff8d40b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:35274/0 (socket says 172.21.15.90:35274) 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2000640 1 -- 172.21.15.90:0/2176692287 learned_addr learned my addr 172.21.15.90:0/2176692287 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2a00640 1 -- 172.21.15.90:0/2176692287 >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 msgr2=0x7ff8d40ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:54.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2a00640 1 --2- 172.21.15.90:0/2176692287 >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 0x7ff8d40ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d1600640 1 -- 172.21.15.90:0/2176692287 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449100745 0 0) 0x7ff8d4160350 con 0x7ff8d40d2680 2024-03-31T15:53:54.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2a00640 1 -- 172.21.15.90:0/2176692287 >> v1:172.21.15.102:6789/0 conn(0x7ff8d40d2680 legacy=0x7ff8d40b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d2a00640 1 -- 172.21.15.90:0/2176692287 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d4160910 con 0x7ff8d415c120 2024-03-31T15:53:54.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.395+0000 7ff8d3400640 1 --2- 172.21.15.90:0/2176692287 >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 0x7ff8d40ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:53:54.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d2a00640 1 --2- 172.21.15.90:0/2176692287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 0x7ff8d415fd00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097c0 tx=0x7ff8c401ce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff883ba29425c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d1600640 1 -- 172.21.15.90:0/2176692287 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40b0070 con 0x7ff8d415c120 2024-03-31T15:53:54.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d1600640 1 -- 172.21.15.90:0/2176692287 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c40ab600 con 0x7ff8d415c120 2024-03-31T15:53:54.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d1600640 1 -- 172.21.15.90:0/2176692287 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40ab9d0 con 0x7ff8d415c120 2024-03-31T15:53:54.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 -- 172.21.15.90:0/2176692287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 msgr2=0x7ff8d415fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 --2- 172.21.15.90:0/2176692287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 0x7ff8d415fd00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097c0 tx=0x7ff8c401ce70 comp rx=0 tx=0).stop 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 -- 172.21.15.90:0/2176692287 shutdown_connections 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 --2- 172.21.15.90:0/2176692287 >> v2:172.21.15.102:3300/0 conn(0x7ff8d40cca30 0x7ff8d40ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 --2- 172.21.15.90:0/2176692287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d415c120 0x7ff8d415fd00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 -- 172.21.15.90:0/2176692287 >> 172.21.15.90:0/2176692287 conn(0x7ff8d409aa40 msgr2=0x7ff8d409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 -- 172.21.15.90:0/2176692287 shutdown_connections 2024-03-31T15:53:54.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.399+0000 7ff8d9800640 1 -- 172.21.15.90:0/2176692287 wait complete. 2024-03-31T15:53:54.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 Processor -- start 2024-03-31T15:53:54.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- start start 2024-03-31T15:53:54.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d40cca30 0x7ff8d4155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8d415c120 0x7ff8d414a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40b2f70 con 0x7ff8d40d2680 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d40b2df0 con 0x7ff8d415c120 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d4169040 con 0x7ff8d40cca30 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43128/0 (socket says 172.21.15.90:43128) 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d40cca30 0x7ff8d4155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 -- 172.21.15.90:0/701943883 learned_addr learned my addr 172.21.15.90:0/701943883 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d3400640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8d415c120 0x7ff8d414a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 -- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d40cca30 msgr2=0x7ff8d4155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d40cca30 0x7ff8d4155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 -- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8d415c120 msgr2=0x7ff8d414a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8d415c120 0x7ff8d414a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 -- 172.21.15.90:0/701943883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d40c7f60 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d2000640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c480 tx=0x7ff8c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c00074e0 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c00047f0 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0004be0 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d40c8240 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.403+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d40c8800 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d40cfd90 con 0x7ff8d40d2680 2024-03-31T15:53:54.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff8c0011070 con 0x7ff8d40d2680 2024-03-31T15:53:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8caa00640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 0x7ff8b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff8c0086940 con 0x7ff8d40d2680 2024-03-31T15:53:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8d2a00640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 0x7ff8b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:54.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.407+0000 7ff8d2a00640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 0x7ff8b406a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ad000 tx=0x7ff8c4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:54.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.411+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff8c0056a90 con 0x7ff8d40d2680 2024-03-31T15:53:54.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8d40686d0 con 0x7ff8d40d2680 2024-03-31T15:53:54.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8caa00640 1 -- 172.21.15.90:0/701943883 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7ff8c0056390 con 0x7ff8d40d2680 2024-03-31T15:53:54.569 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:54.569 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 msgr2=0x7ff8b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 0x7ff8b406a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ad000 tx=0x7ff8c4004740 comp rx=0 tx=0).stop 2024-03-31T15:53:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 msgr2=0x7ff8d414a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c480 tx=0x7ff8c000c880 comp rx=0 tx=0).stop 2024-03-31T15:53:54.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 shutdown_connections 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8b4068380 0x7ff8b406a840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8d415c120 0x7ff8d414a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff8d40d2680 0x7ff8d414a040 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 --2- 172.21.15.90:0/701943883 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff8d40cca30 0x7ff8d4155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 >> 172.21.15.90:0/701943883 conn(0x7ff8d409aa40 msgr2=0x7ff8d41554e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.575+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 shutdown_connections 2024-03-31T15:53:54.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:54.579+0000 7ff8d9800640 1 -- 172.21.15.90:0/701943883 wait complete. 2024-03-31T15:53:54.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:53:47.115176+0000', 'modified': '2024-03-31T15:53:52.163102+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6835/3620557044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 3620557044}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 3620557044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-03-31T15:53:54.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:53:54.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:53:55.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 Processor -- start 2024-03-31T15:53:55.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 -- start start 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f940c06ff00 0x7f940c0add80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f940c06e4e0 con 0x7f940c15c120 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f940c06e7c0 con 0x7f940c06b5f0 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9410a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f940c06f0f0 con 0x7f940c06ff00 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9409e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f940c15c120 0x7f940c06f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57816/0 (socket says 172.21.15.90:57816) 2024-03-31T15:53:55.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9409e00640 1 -- 172.21.15.90:0/286292732 learned_addr learned my addr 172.21.15.90:0/286292732 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:55.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9403e00640 1 --2- 172.21.15.90:0/286292732 >> v2:172.21.15.102:3300/0 conn(0x7f940c06ff00 0x7f940c0add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1921595203 0 0) 0x7f940c06e4e0 con 0x7f940c15c120 2024-03-31T15:53:55.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93f0003530 con 0x7f940c15c120 2024-03-31T15:53:55.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9403e00640 1 -- 172.21.15.90:0/286292732 >> v1:172.21.15.102:6789/0 conn(0x7f940c06b5f0 legacy=0x7f940c15ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9403e00640 1 -- 172.21.15.90:0/286292732 >> v1:172.21.15.90:6789/0 conn(0x7f940c15c120 legacy=0x7f940c06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3623789090 0 0) 0x7f93f0003530 con 0x7f940c15c120 2024-03-31T15:53:55.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.139+0000 7f9403e00640 1 -- 172.21.15.90:0/286292732 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f940c06eaa0 con 0x7f940c06ff00 2024-03-31T15:53:55.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9403e00640 1 --2- 172.21.15.90:0/286292732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c0add80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93f40097c0 tx=0x7f93f401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e1f70432b8a1319 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f40b1070 con 0x7f940c06ff00 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f40ac410 con 0x7f940c06ff00 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9409400640 1 -- 172.21.15.90:0/286292732 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f40ac7e0 con 0x7f940c06ff00 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 -- 172.21.15.90:0/286292732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 msgr2=0x7f940c0add80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 --2- 172.21.15.90:0/286292732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c0add80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f93f40097c0 tx=0x7f93f401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 -- 172.21.15.90:0/286292732 shutdown_connections 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 --2- 172.21.15.90:0/286292732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c0add80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 -- 172.21.15.90:0/286292732 >> 172.21.15.90:0/286292732 conn(0x7f940c06d140 msgr2=0x7f940c0cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 -- 172.21.15.90:0/286292732 shutdown_connections 2024-03-31T15:53:55.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.143+0000 7f9410a00640 1 -- 172.21.15.90:0/286292732 wait complete. 2024-03-31T15:53:55.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 Processor -- start 2024-03-31T15:53:55.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- start start 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f940c15c120 0x7f940c0b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f940c160c10 con 0x7f940c06b5f0 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f940c160a90 con 0x7f940c06ff00 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f940c16a8f0 con 0x7f940c15c120 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43140/0 (socket says 172.21.15.90:43140) 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 -- 172.21.15.90:0/3207794378 learned_addr learned my addr 172.21.15.90:0/3207794378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 -- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f940c15c120 msgr2=0x7f940c0b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f940c15c120 0x7f940c0b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 -- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 msgr2=0x7f940c155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9403e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 -- 172.21.15.90:0/3207794378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f940c0ce770 con 0x7f940c06b5f0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9403e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9409e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00c410 tx=0x7f93fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9402000640 1 -- 172.21.15.90:0/3207794378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc017070 con 0x7f940c06b5f0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9402000640 1 -- 172.21.15.90:0/3207794378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93fc0074e0 con 0x7f940c06b5f0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9402000640 1 -- 172.21.15.90:0/3207794378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0059f0 con 0x7f940c06b5f0 2024-03-31T15:53:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f940c155cd0 con 0x7f940c06b5f0 2024-03-31T15:53:55.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.147+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f940c15a540 con 0x7f940c06b5f0 2024-03-31T15:53:55.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9402000640 1 -- 172.21.15.90:0/3207794378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f93fc01a070 con 0x7f940c06b5f0 2024-03-31T15:53:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9402000640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 0x7f93e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9402000640 1 -- 172.21.15.90:0/3207794378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f93fc086060 con 0x7f940c06b5f0 2024-03-31T15:53:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 Processor -- start 2024-03-31T15:53:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9403e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 0x7f93e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 -- start start 2024-03-31T15:53:55.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9403e00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 0x7f93e806a870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f93f40ae000 tx=0x7f93f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915b4a0 0x55941915b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941910f0b0 0x55941910f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559418f61950 con 0x55941915bdc0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559418efe990 con 0x55941915b4a0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.151+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x559418f5d900 con 0x55941910f0b0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43154/0 (socket says 172.21.15.90:43154) 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 -- 172.21.15.90:0/1594112371 learned_addr learned my addr 172.21.15.90:0/1594112371 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9409e00640 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915b4a0 0x55941915b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f940a800640 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941910f0b0 0x55941910f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 -- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941910f0b0 msgr2=0x55941910f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941910f0b0 0x55941910f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 -- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915b4a0 msgr2=0x55941915b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915b4a0 0x55941915b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 -- 172.21.15.90:0/1594112371 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559418fa6100 con 0x55941915bdc0 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9403e00640 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93f401ca10 tx=0x7f93f4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce38773a8179df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f93faa00640 1 -- 172.21.15.90:0/1594112371 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f40acd20 con 0x55941915bdc0 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f93faa00640 1 -- 172.21.15.90:0/1594112371 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f40c7070 con 0x55941915bdc0 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f93faa00640 1 -- 172.21.15.90:0/1594112371 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f40b8b30 con 0x55941915bdc0 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- 172.21.15.90:0/1594112371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 msgr2=0x55941910eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93f401ca10 tx=0x7f93f4004740 comp rx=0 tx=0).stop 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- 172.21.15.90:0/1594112371 shutdown_connections 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941910f0b0 0x55941910f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915bdc0 0x55941910eb70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- 172.21.15.90:0/1594112371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915b4a0 0x55941915b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- 172.21.15.90:0/1594112371 >> 172.21.15.90:0/1594112371 conn(0x559418ff9b80 msgr2=0x55941913e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- 172.21.15.90:0/1594112371 shutdown_connections 2024-03-31T15:53:55.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- 172.21.15.90:0/1594112371 wait complete. 2024-03-31T15:53:55.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- start start 2024-03-31T15:53:55.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 Processor -- start 2024-03-31T15:53:55.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941911c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x55941911c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559418f60c60 con 0x55941910f0b0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559418f1f3e0 con 0x55941915bdc0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x559418f6e060 con 0x55941915b4a0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.155+0000 7f9413ede000 10 client.0 _ll_register_callbacks cb 0x559418fc20f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43164/0 (socket says 172.21.15.90:43164) 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 -- 172.21.15.90:0/2632843649 learned_addr learned my addr 172.21.15.90:0/2632843649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9403e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941911c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f940a800640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x55941911c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 -- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 msgr2=0x55941911c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941911c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 -- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 msgr2=0x55941911c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x55941911c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 -- 172.21.15.90:0/2632843649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559418f2b8c0 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f940a800640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x55941911c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9409e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93fc055c00 tx=0x7f93fc05a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2398acd66a941fab server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc06e200 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93f40c7070 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f93fc013df0 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f93fc0550c0 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 10 client.4419 fetch_fsmap learned FSMap version 7 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x559418ee9e80 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f40c7070 con 0x55941910f0b0 2024-03-31T15:53:55.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f93fc055260 con 0x55941910f0b0 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 10 client.4419 fetch_fsmap finished waiting for FSMap version 7 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 10 client.4419 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 20 client.4419 populate_metadata read hostname 'smithi090' 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941911d120 0x55941916da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 4 client.4419 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 --> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559418efcbc0 con 0x55941911d120 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9403e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941911d120 0x55941916da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9403e00640 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941911d120 0x55941916da20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c07c130b223d70 server_cookie=77d9366c4ba5040b in_seq=0 out_seq=0 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 10 client.4419 ms_handle_connect on v2:172.21.15.90:6834/3620557044 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 1 -- 172.21.15.90:0/2632843649 <== mds.0 v2:172.21.15.90:6834/3620557044 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559418efcbc0 con 0x55941911d120 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 10 client.4419 handle_command_reply: tid=0 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 client.4419 shutdown 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 msgr2=0x55941911be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f93fc055c00 tx=0x7f93fc05a160 comp rx=0 tx=0).stop 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 shutdown_connections 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941911d120 0x55941916da20 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x55941911c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941911c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f93ef400640 0 client.4419 ms_handle_reset on v2:172.21.15.90:6834/3620557044 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 --2- 172.21.15.90:0/2632843649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941911be00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 >> 172.21.15.90:0/2632843649 conn(0x559418ff9b80 msgr2=0x55941910e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 shutdown_connections 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 1 -- 172.21.15.90:0/2632843649 wait complete. 2024-03-31T15:53:55.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.159+0000 7f9413ede000 20 client.4419 trim_cache size 0 max 16384 2024-03-31T15:53:55.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 Processor -- start 2024-03-31T15:53:55.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 -- start start 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941910e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x559419159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f940c205040 con 0x55941910f0b0 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f940c204060 con 0x55941915bdc0 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f940c203a10 con 0x55941915b4a0 2024-03-31T15:53:55.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43166/0 (socket says 172.21.15.90:43166) 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 -- 172.21.15.90:0/470946401 learned_addr learned my addr 172.21.15.90:0/470946401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9403e00640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941910e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f940a800640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x559419159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 -- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 msgr2=0x55941910e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941910e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 -- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 msgr2=0x559419159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x559419159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 -- 172.21.15.90:0/470946401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559418f1c490 con 0x55941910f0b0 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9403e00640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941910e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9409e00640 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f93fc011000 tx=0x7f93fc058000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0c525a7db68b52 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f93fa000640 1 -- 172.21.15.90:0/470946401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc055260 con 0x55941910f0b0 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f93fa000640 1 -- 172.21.15.90:0/470946401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93fc05c340 con 0x55941910f0b0 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f93fa000640 1 -- 172.21.15.90:0/470946401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0519c0 con 0x55941910f0b0 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 -- 172.21.15.90:0/470946401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 msgr2=0x55941910e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.179+0000 7f9413ede000 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f93fc011000 tx=0x7f93fc058000 comp rx=0 tx=0).stop 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/470946401 shutdown_connections 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941915bdc0 0x559419159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915b4a0 0x55941910e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- 172.21.15.90:0/470946401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941910f0b0 0x55941910e2e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/470946401 >> 172.21.15.90:0/470946401 conn(0x559418ff9b80 msgr2=0x559419168a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/470946401 shutdown_connections 2024-03-31T15:53:55.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/470946401 wait complete. 2024-03-31T15:53:55.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- start start 2024-03-31T15:53:55.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 Processor -- start 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941910f0b0 0x559419170190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915bdc0 0x5594191c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x559418ee5b10 con 0x55941915b4a0 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559418f65f80 con 0x55941910f0b0 2024-03-31T15:53:55.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x559418f3aef0 con 0x55941915bdc0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 10 client.0 _ll_register_callbacks cb 0x7f93fc0558a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43178/0 (socket says 172.21.15.90:43178) 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 -- 172.21.15.90:0/2713879070 learned_addr learned my addr 172.21.15.90:0/2713879070 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9409e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941910f0b0 0x559419170190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f940a800640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915bdc0 0x5594191c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 -- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915bdc0 msgr2=0x5594191c84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915bdc0 0x5594191c84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 -- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941910f0b0 msgr2=0x559419170190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941910f0b0 0x559419170190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 -- 172.21.15.90:0/2713879070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559418f1ae80 con 0x55941915b4a0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9403e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f93f401c980 tx=0x7f93f401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01def2b3a0ce545 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:53:55.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f40d3070 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93fc05c340 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f93f40d4b50 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f93f40ac410 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 10 client.4425 fetch_fsmap learned FSMap version 7 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x559418f08270 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93fc05c340 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f93f4004040 con 0x55941915b4a0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 10 client.4425 fetch_fsmap finished waiting for FSMap version 7 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 10 client.4425 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 20 client.4425 populate_metadata read hostname 'smithi090' 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941916da20 0x55941916de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 4 client.4425 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 --> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] -- command(tid 0: {"prefix": "status"}) -- 0x559418f6ee90 con 0x55941916da20 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.183+0000 7f9409e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941916da20 0x55941916de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f93ef400640 10 client.4425 ms_handle_connect on v2:172.21.15.90:6834/3620557044 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9409e00640 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941916da20 0x55941916de00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90938e27cc67ef68 server_cookie=2b999b4cb4dfb9a2 in_seq=0 out_seq=0 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f93ef400640 1 -- 172.21.15.90:0/2713879070 <== mds.0 v2:172.21.15.90:6834/3620557044 1 ==== command_reply(tid 0: 0 ) ==== 8+0+317 (crc 0 0 0) 0x559418f6ee90 con 0x55941916da20 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f93ef400640 10 client.4425 handle_command_reply: tid=0 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 client.4425 shutdown 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 msgr2=0x5594191c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f93f401c980 tx=0x7f93f401ace0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 shutdown_connections 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x55941916da20 0x55941916de00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55941915bdc0 0x5594191c84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55941915b4a0 0x5594191c7f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 --2- 172.21.15.90:0/2713879070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55941910f0b0 0x559419170190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 >> 172.21.15.90:0/2713879070 conn(0x559418ff9b80 msgr2=0x559419168a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f93ef400640 0 client.4425 ms_handle_reset on v2:172.21.15.90:6834/3620557044 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 shutdown_connections 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 1 -- 172.21.15.90:0/2713879070 wait complete. 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9413ede000 20 client.4425 trim_cache size 0 max 16384 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "0135ca30-606a-448f-b61f-3ca22bf9710a", 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4293, 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-31T15:53:55.187 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 7.439892671, 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 7, 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 25, 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 25, 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 12.118491039 2024-03-31T15:53:55.188 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 msgr2=0x7f93e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 0x7f93e806a870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f93f40ae000 tx=0x7f93f4008040 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 msgr2=0x7f940c0cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00c410 tx=0x7f93fc00c8d0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 shutdown_connections 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93e80683b0 0x7f93e806a870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f940c15c120 0x7f940c0b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f940c06ff00 0x7f940c155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 --2- 172.21.15.90:0/3207794378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f940c06b5f0 0x7f940c0cd150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 >> 172.21.15.90:0/3207794378 conn(0x7f940c06d140 msgr2=0x7f940c159920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 shutdown_connections 2024-03-31T15:53:55.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.187+0000 7f9410a00640 1 -- 172.21.15.90:0/3207794378 wait complete. 2024-03-31T15:53:55.201 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:53:55.201 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:55.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:53:55.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.343+0000 7f438b200640 1 Processor -- start 2024-03-31T15:53:55.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 -- start start 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f438406d5f0 0x7f438415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f438415f780 con 0x7f438415e1b0 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f438406daa0 con 0x7f438406d5f0 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f438b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43841622b0 con 0x7f438406c7b0 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f438406d5f0 0x7f438415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f438406d5f0 0x7f438415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43188/0 (socket says 172.21.15.90:43188) 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 -- 172.21.15.90:0/4070687039 learned_addr learned my addr 172.21.15.90:0/4070687039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f437be00640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 -- 172.21.15.90:0/4070687039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 msgr2=0x7f438406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 -- 172.21.15.90:0/4070687039 >> v1:172.21.15.102:6789/0 conn(0x7f438415e1b0 legacy=0x7f4384162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:55.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.347+0000 7f4389200640 1 -- 172.21.15.90:0/4070687039 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438415f1c0 con 0x7f438406d5f0 2024-03-31T15:53:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f4389200640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438406d5f0 0x7f438415e590 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f437c009e50 tx=0x7f437c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b2c24f371c444b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f4383400640 1 -- 172.21.15.90:0/4070687039 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c0a57b0 con 0x7f438406d5f0 2024-03-31T15:53:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f4383400640 1 -- 172.21.15.90:0/4070687039 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c0a5950 con 0x7f438406d5f0 2024-03-31T15:53:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f4383400640 1 -- 172.21.15.90:0/4070687039 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c0a5d20 con 0x7f438406d5f0 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 -- 172.21.15.90:0/4070687039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438406d5f0 msgr2=0x7f438415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438406d5f0 0x7f438415e590 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f437c009e50 tx=0x7f437c01c930 comp rx=0 tx=0).stop 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 -- 172.21.15.90:0/4070687039 shutdown_connections 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438406d5f0 0x7f438415e590 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 --2- 172.21.15.90:0/4070687039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 -- 172.21.15.90:0/4070687039 >> 172.21.15.90:0/4070687039 conn(0x7f438409aa40 msgr2=0x7f438409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 -- 172.21.15.90:0/4070687039 shutdown_connections 2024-03-31T15:53:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.351+0000 7f438b200640 1 -- 172.21.15.90:0/4070687039 wait complete. 2024-03-31T15:53:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 Processor -- start 2024-03-31T15:53:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 -- start start 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438415b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f438406d5f0 0x7f43840afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43840d2940 con 0x7f438415e1b0 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43840d27c0 con 0x7f438406d5f0 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f438b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43840d2ac0 con 0x7f438406c7b0 2024-03-31T15:53:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f4389200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f4389200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43200/0 (socket says 172.21.15.90:43200) 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f4389200640 1 -- 172.21.15.90:0/876565214 learned_addr learned my addr 172.21.15.90:0/876565214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f4383e00640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438415b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.355+0000 7f437be00640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f438406d5f0 0x7f43840afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 -- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 msgr2=0x7f438415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438415b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 -- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f438406d5f0 msgr2=0x7f43840afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f438406d5f0 0x7f43840afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 -- 172.21.15.90:0/876565214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43840b5040 con 0x7f438415e1b0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4389200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f437c01afe0 tx=0x7f437c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c0b8670 con 0x7f438415e1b0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c0be070 con 0x7f438415e1b0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43840b5320 con 0x7f438415e1b0 2024-03-31T15:53:55.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c0b8a30 con 0x7f438415e1b0 2024-03-31T15:53:55.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43840b58e0 con 0x7f438415e1b0 2024-03-31T15:53:55.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43840686d0 con 0x7f438415e1b0 2024-03-31T15:53:55.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.359+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f437c0b8bd0 con 0x7f438415e1b0 2024-03-31T15:53:55.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.363+0000 7f4380c00640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 0x7f436006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:55.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.363+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f437c121db0 con 0x7f438415e1b0 2024-03-31T15:53:55.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.363+0000 7f4383e00640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 0x7f436006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:55.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.363+0000 7f4383e00640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 0x7f436006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4374007fd0 tx=0x7f4374007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:55.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.367+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f437c0165e0 con 0x7f438415e1b0 2024-03-31T15:53:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4384169f90 con 0x7f438415e1b0 2024-03-31T15:53:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f4380c00640 1 -- 172.21.15.90:0/876565214 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f437c0f1950 con 0x7f438415e1b0 2024-03-31T15:53:55.552 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T15:53:55.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 msgr2=0x7f436006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 0x7f436006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4374007fd0 tx=0x7f4374007e40 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 msgr2=0x7f438415cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f437c01afe0 tx=0x7f437c0196d0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 shutdown_connections 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4360068380 0x7f436006a840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f438415e1b0 0x7f438415cf10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f438406d5f0 0x7f43840afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 --2- 172.21.15.90:0/876565214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f438406c7b0 0x7f438415b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 >> 172.21.15.90:0/876565214 conn(0x7f438409aa40 msgr2=0x7f43840b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 shutdown_connections 2024-03-31T15:53:55.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55.559+0000 7f438b200640 1 -- 172.21.15.90:0/876565214 wait complete. 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:55.581 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-31T15:53:55.582 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:53:55.582 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:53:55.582 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-31T15:53:55.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:55.582 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:53:55.591 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:53:55.592 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:53:55.593 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T15:53:55.593 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T15:53:55.594 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:53:55.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:53:55.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:55.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:55.748 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:53:55.768 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:53:55.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:55.770 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:53:55.849 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:53:55.850 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:53:55.850 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:53:55.850 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:53:55.850 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:53:55.850 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:53:55.850 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:53:55.850 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:53:55.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:53:55.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:53:55.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:53:55.851 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:53:55.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:53:55.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:55.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:55.983 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:53:55.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:55.993 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-31T15:53:56.042 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:53:56.043 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:53:56.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:56.043 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-31T15:53:56.043 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:53:56.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:53:56.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:56.166 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-31T15:53:56.166 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:53:56.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:53:56.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:56.345 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:53:56.345 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:53:56.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:53:56.345 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T15:53:56.345 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:53:56.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:53:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:56.429 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-31T15:53:56.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:53:56.429 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:56.437 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-31T15:53:56.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:53:56.437 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:56.490 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:53:56.550 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:53:56.600 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:53:56.601 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:53:56.602 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:53:56.602 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:53:56.602 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:56.602 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:56.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:56.603 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:53:56.654 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:53:56.654 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-31T15:53:56.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:53:56.724 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:53:56.776 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:53:56.777 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:56.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:56.779 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:53:56.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:2024-03-31T15:53:56.803+0000 7f444d1d7580 -1 init, newargv = 0x557caf6e2250 newargc=15 2024-03-31T15:53:56.798 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[33371]: starting ceph client 2024-03-31T15:53:56.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[33371]: starting fuse 2024-03-31T15:53:57.908 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:53:57.928 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:53:57.983 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:53:57.983 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:53:57.984 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:53:57.985 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:57.986 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:53:57.986 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:57.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:57.987 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:53:58.038 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:53:58.038 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:53:58.039 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.admin")) 2024-03-31T15:53:58.040 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:53:58.157 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.admin.33371.asok 2024-03-31T15:53:58.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:58.170 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.33371.asok 2024-03-31T15:53:58.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:58.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.33371.asok status 2024-03-31T15:53:58.337 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "admin", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "33371", 2024-03-31T15:53:58.338 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4434, 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:53:58.339 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4434 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2947767195 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:53:58.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2947767195 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4434 192.168.0.1:0/2947767195", 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/2947767195", 2024-03-31T15:53:58.341 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 7, 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 25, 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:53:58.342 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:53:58.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:58.353 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:53:58.353 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:58.363 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:53:58.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:58.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:53:58.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:53:58.427 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-31T15:53:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-31T15:53:58.538 INFO:teuthology.orchestra.run.smithi090.stdout:# file: . 2024-03-31T15:53:58.538 INFO:teuthology.orchestra.run.smithi090.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-31T15:53:58.538 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:58.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:53:58.545 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:53:58.545 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:53:58.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:53:58.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 Processor -- start 2024-03-31T15:53:58.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 -- start start 2024-03-31T15:53:58.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f109815fd00 0x7f10980d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f10980cca30 0x7f10980cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1098160350 con 0x7f109815c120 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1098160bf0 con 0x7f109815fd00 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f109da00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f10980cd3a0 con 0x7f10980cca30 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f109815fd00 0x7f10980d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f109815fd00 0x7f10980d2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43208/0 (socket says 172.21.15.90:43208) 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 -- 172.21.15.90:0/1936340917 learned_addr learned my addr 172.21.15.90:0/1936340917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:58.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f1097400640 1 --2- 172.21.15.90:0/1936340917 >> v2:172.21.15.102:3300/0 conn(0x7f10980cca30 0x7f10980cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 -- 172.21.15.90:0/1936340917 >> v2:172.21.15.102:3300/0 conn(0x7f10980cca30 msgr2=0x7f10980cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 --2- 172.21.15.90:0/1936340917 >> v2:172.21.15.102:3300/0 conn(0x7f10980cca30 0x7f10980cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 -- 172.21.15.90:0/1936340917 >> v1:172.21.15.102:6789/0 conn(0x7f109815c120 legacy=0x7f10980b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.735+0000 7f108fe00640 1 -- 172.21.15.90:0/1936340917 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1098160630 con 0x7f109815fd00 2024-03-31T15:53:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f108fe00640 1 --2- 172.21.15.90:0/1936340917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10980d2680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10980d3000 tx=0x7f108001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1f338ce4676063 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f1096000640 1 -- 172.21.15.90:0/1936340917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10800b1070 con 0x7f109815fd00 2024-03-31T15:53:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f1096000640 1 -- 172.21.15.90:0/1936340917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10800ac600 con 0x7f109815fd00 2024-03-31T15:53:58.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f1096000640 1 -- 172.21.15.90:0/1936340917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10800ac9d0 con 0x7f109815fd00 2024-03-31T15:53:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 -- 172.21.15.90:0/1936340917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 msgr2=0x7f10980d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 --2- 172.21.15.90:0/1936340917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10980d2680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f10980d3000 tx=0x7f108001c490 comp rx=0 tx=0).stop 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 -- 172.21.15.90:0/1936340917 shutdown_connections 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 --2- 172.21.15.90:0/1936340917 >> v2:172.21.15.102:3300/0 conn(0x7f10980cca30 0x7f10980cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 --2- 172.21.15.90:0/1936340917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10980d2680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 -- 172.21.15.90:0/1936340917 >> 172.21.15.90:0/1936340917 conn(0x7f109809aa40 msgr2=0x7f109809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 -- 172.21.15.90:0/1936340917 shutdown_connections 2024-03-31T15:53:58.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.739+0000 7f109da00640 1 -- 172.21.15.90:0/1936340917 wait complete. 2024-03-31T15:53:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 Processor -- start 2024-03-31T15:53:58.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- start start 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 0x7f10981514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 0x7f1098154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f10980b2f70 con 0x7f109815fd00 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10980b2df0 con 0x7f10980cca30 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1098169040 con 0x7f109815c120 2024-03-31T15:53:58.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43210/0 (socket says 172.21.15.90:43210) 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 -- 172.21.15.90:0/2009140950 learned_addr learned my addr 172.21.15.90:0/2009140950 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1096a00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 0x7f10981514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f108fe00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 0x7f1098154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 -- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 msgr2=0x7f1098154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 0x7f1098154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 -- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 msgr2=0x7f10981514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 0x7f10981514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 -- 172.21.15.90:0/2009140950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10981519e0 con 0x7f109815fd00 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f108fe00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 0x7f1098154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1096a00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 0x7f10981514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f1097400640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f109000efc0 tx=0x7f109000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090018020 con 0x7f109815fd00 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1090014480 con 0x7f109815fd00 2024-03-31T15:53:58.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109806df30 con 0x7f109815fd00 2024-03-31T15:53:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090014870 con 0x7f109815fd00 2024-03-31T15:53:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109806d970 con 0x7f109815fd00 2024-03-31T15:53:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.743+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10980c6010 con 0x7f109815fd00 2024-03-31T15:53:58.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.747+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f109001b020 con 0x7f109815fd00 2024-03-31T15:53:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.747+0000 7f108ea00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 0x7f107406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.747+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1090087770 con 0x7f109815fd00 2024-03-31T15:53:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.747+0000 7f1096a00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 0x7f107406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:58.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.747+0000 7f1096a00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 0x7f107406a870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1088001670 tx=0x7f1088009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:58.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.751+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1090022070 con 0x7f109815fd00 2024-03-31T15:53:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.911+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10980686d0 con 0x7f109815fd00 2024-03-31T15:53:58.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.911+0000 7f108ea00640 1 -- 172.21.15.90:0/2009140950 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f1090010040 con 0x7f109815fd00 2024-03-31T15:53:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:53:58.906 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:53:58.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 msgr2=0x7f107406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 0x7f107406a870 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1088001670 tx=0x7f1088009040 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 msgr2=0x7f10981548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f109000efc0 tx=0x7f109000c490 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 shutdown_connections 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f10740683b0 0x7f107406a870 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f109815fd00 0x7f10981548a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f109815c120 0x7f1098154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 --2- 172.21.15.90:0/2009140950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980cca30 0x7f10981514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 >> 172.21.15.90:0/2009140950 conn(0x7f109809aa40 msgr2=0x7f109806ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 shutdown_connections 2024-03-31T15:53:58.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:58.915+0000 7f109da00640 1 -- 172.21.15.90:0/2009140950 wait complete. 2024-03-31T15:53:58.927 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:53:58.928 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:53:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 Processor -- start 2024-03-31T15:53:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 -- start start 2024-03-31T15:53:59.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f80aeb00 0x7f64f80aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64f8071750 0x7f64f8071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f806e9c0 con 0x7f64f815c290 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f64f806de40 con 0x7f64f8071750 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.071+0000 7f64ffa00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f64f806e120 con 0x7f64f80aeb00 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fd000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f64f815c290 0x7f64f80ace90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57822/0 (socket says 172.21.15.90:57822) 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fd000640 1 -- 172.21.15.90:0/3695192722 learned_addr learned my addr 172.21.15.90:0/3695192722 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 --2- 172.21.15.90:0/3695192722 >> v2:172.21.15.102:3300/0 conn(0x7f64f8071750 0x7f64f8071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64f7e00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f80aeb00 0x7f64f80aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64f7400640 1 -- 172.21.15.90:0/3695192722 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680368203 0 0) 0x7f64f806e9c0 con 0x7f64f815c290 2024-03-31T15:53:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64f7400640 1 -- 172.21.15.90:0/3695192722 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64e0003530 con 0x7f64f815c290 2024-03-31T15:53:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 -- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f80aeb00 msgr2=0x7f64f80aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f80aeb00 0x7f64f80aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 -- 172.21.15.90:0/3695192722 >> v1:172.21.15.90:6789/0 conn(0x7f64f815c290 legacy=0x7f64f80ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 -- 172.21.15.90:0/3695192722 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f806e400 con 0x7f64f8071750 2024-03-31T15:53:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64fda00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f8071750 0x7f64f8071be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009d90 tx=0x7f64f001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63818a110d1a4a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.075+0000 7f64f7400640 1 -- 172.21.15.90:0/3695192722 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f00b2070 con 0x7f64f8071750 2024-03-31T15:53:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64f7400640 1 -- 172.21.15.90:0/3695192722 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f00a5830 con 0x7f64f8071750 2024-03-31T15:53:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64f7400640 1 -- 172.21.15.90:0/3695192722 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f00a5c00 con 0x7f64f8071750 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f8071750 msgr2=0x7f64f8071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f8071750 0x7f64f8071be0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009d90 tx=0x7f64f001c970 comp rx=0 tx=0).stop 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- 172.21.15.90:0/3695192722 shutdown_connections 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f8071750 0x7f64f8071be0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- 172.21.15.90:0/3695192722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f80aeb00 0x7f64f80aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- 172.21.15.90:0/3695192722 >> 172.21.15.90:0/3695192722 conn(0x7f64f80b18b0 msgr2=0x7f64f80b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- 172.21.15.90:0/3695192722 shutdown_connections 2024-03-31T15:53:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- 172.21.15.90:0/3695192722 wait complete. 2024-03-31T15:53:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 Processor -- start 2024-03-31T15:53:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- start start 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 0x7f64f8152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 0x7f64f80cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f64f806db70 con 0x7f64f80aeb00 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f64f806d9f0 con 0x7f64f815c290 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64ffa00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f64f8169000 con 0x7f64f8071750 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64f7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43212/0 (socket says 172.21.15.90:43212) 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64f7e00640 1 -- 172.21.15.90:0/2030385697 learned_addr learned my addr 172.21.15.90:0/2030385697 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.079+0000 7f64fd000640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 0x7f64f8152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64fda00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 0x7f64f80cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 -- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 msgr2=0x7f64f8152ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 0x7f64f8152ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 -- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 msgr2=0x7f64f80cfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 0x7f64f80cfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 -- 172.21.15.90:0/2030385697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f809c450 con 0x7f64f80aeb00 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64fda00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 0x7f64f80cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64fd000640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 0x7f64f8152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f7e00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e930 tx=0x7f64e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4017070 con 0x7f64f80aeb00 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e400ca00 con 0x7f64f80aeb00 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f815f6d0 con 0x7f64f80aeb00 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e400cdb0 con 0x7f64f80aeb00 2024-03-31T15:53:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f815ccc0 con 0x7f64f80aeb00 2024-03-31T15:53:59.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.083+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f8075880 con 0x7f64f80aeb00 2024-03-31T15:53:59.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f64e401a070 con 0x7f64f80aeb00 2024-03-31T15:53:59.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64f4c00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 0x7f64d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f64e4087650 con 0x7f64f80aeb00 2024-03-31T15:53:59.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64fd000640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 0x7f64d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64fd000640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 0x7f64d806a870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005890 tx=0x7f64ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.087+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f64f809c450 con 0x7f64f80aeb00 2024-03-31T15:53:59.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.247+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64f806ebf0 con 0x7f64f80aeb00 2024-03-31T15:53:59.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.247+0000 7f64f4c00640 1 -- 172.21.15.90:0/2030385697 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f64e40578f0 con 0x7f64f80aeb00 2024-03-31T15:53:59.243 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 msgr2=0x7f64d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 0x7f64d806a870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f64ec005890 tx=0x7f64ec009040 comp rx=0 tx=0).stop 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 msgr2=0x7f64f815f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e930 tx=0x7f64e400edf0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 shutdown_connections 2024-03-31T15:53:59.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f64d80683b0 0x7f64d806a870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f815c290 0x7f64f80cfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f64f80aeb00 0x7f64f815f190 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 --2- 172.21.15.90:0/2030385697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f64f8071750 0x7f64f8152ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 >> 172.21.15.90:0/2030385697 conn(0x7f64f80b18b0 msgr2=0x7f64f80ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 shutdown_connections 2024-03-31T15:53:59.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.251+0000 7f64ffa00640 1 -- 172.21.15.90:0/2030385697 wait complete. 2024-03-31T15:53:59.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:53:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.411+0000 7f33a3e00640 1 Processor -- start 2024-03-31T15:53:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.411+0000 7f33a3e00640 1 -- start start 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33a406c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a40b3f80 con 0x7f33a40cc8d0 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a40b4260 con 0x7f33a406e970 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a3e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f33a40b3d30 con 0x7f33a415e540 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f33a40cc8d0 0x7f33a40cdb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57832/0 (socket says 172.21.15.90:57832) 2024-03-31T15:53:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 -- 172.21.15.90:0/2270267137 learned_addr learned my addr 172.21.15.90:0/2270267137 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461410187 0 0) 0x7f33a40b3f80 con 0x7f33a40cc8d0 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2a00640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33a406c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3388003530 con 0x7f33a40cc8d0 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454699698 0 0) 0x7f3388003530 con 0x7f33a40cc8d0 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 msgr2=0x7f33a406c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33a406c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 >> v1:172.21.15.102:6789/0 conn(0x7f33a406e970 legacy=0x7f33a40ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a40b4540 con 0x7f33a40cc8d0 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f3390001680 con 0x7f33a40cc8d0 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 >> v1:172.21.15.90:6789/0 conn(0x7f33a40cc8d0 legacy=0x7f33a40cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33880039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3388003f70 0x7f3388004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388004d50 0x7f33880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33a40b3d30 con 0x7f3388003f70 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f338800bab0 con 0x7f3388004d50 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f338800dec0 con 0x7f33a415e540 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3388003f70 0x7f3388004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2a00640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33880039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a3400640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388004d50 0x7f33880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 -- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 msgr2=0x7f33880039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a415e540 0x7f33880039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 -- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388004d50 msgr2=0x7f33880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388004d50 0x7f33880051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.415+0000 7f33a2000640 1 -- 172.21.15.90:0/2270267137 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3388003530 con 0x7f3388003f70 2024-03-31T15:53:59.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a2000640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3388003f70 0x7f3388004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f339000c240 tx=0x7f339001f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cb7c59eb75b3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390007d20 con 0x7f3388003f70 2024-03-31T15:53:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3390005ae0 con 0x7f3388003f70 2024-03-31T15:53:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a1600640 1 -- 172.21.15.90:0/2270267137 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390007070 con 0x7f3388003f70 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 -- 172.21.15.90:0/2270267137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3388003f70 msgr2=0x7f3388004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2270267137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3388003f70 0x7f3388004810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f339000c240 tx=0x7f339001f3f0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3400640 1 -- 172.21.15.90:0/2270267137 reap_dead start 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 -- 172.21.15.90:0/2270267137 shutdown_connections 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 -- 172.21.15.90:0/2270267137 >> 172.21.15.90:0/2270267137 conn(0x7f33a40d0390 msgr2=0x7f33a40d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 -- 172.21.15.90:0/2270267137 shutdown_connections 2024-03-31T15:53:59.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 -- 172.21.15.90:0/2270267137 wait complete. 2024-03-31T15:53:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.419+0000 7f33a3e00640 1 Processor -- start 2024-03-31T15:53:59.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- start start 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388003f70 0x7f33a40ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a4159fd0 0x7f33a415a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33a40b4b00 con 0x7f33a40aee40 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f33a415e920 con 0x7f3388003f70 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f33a40b5020 con 0x7f33a4159fd0 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43240/0 (socket says 172.21.15.90:43240) 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 -- 172.21.15.90:0/2646652026 learned_addr learned my addr 172.21.15.90:0/2646652026 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:59.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3400640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a4159fd0 0x7f33a415a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2a00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388003f70 0x7f33a40ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 -- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a4159fd0 msgr2=0x7f33a415a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a4159fd0 0x7f33a415a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 -- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388003f70 msgr2=0x7f33a40ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388003f70 0x7f33a40ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 -- 172.21.15.90:0/2646652026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a40ca080 con 0x7f33a40aee40 2024-03-31T15:53:59.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a2000640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f339000c240 tx=0x7f3388005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390007d20 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33900047f0 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a40ca360 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390004ba0 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a40c9d50 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.423+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a40af2c0 con 0x7f33a40aee40 2024-03-31T15:53:59.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.427+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3390007070 con 0x7f33a40aee40 2024-03-31T15:53:59.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.427+0000 7f339b400640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 0x7f338406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.427+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f33901240a0 con 0x7f33a40aee40 2024-03-31T15:53:59.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.427+0000 7f33a2a00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 0x7f338406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.427+0000 7f33a2a00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 0x7f338406a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3394009be0 tx=0x7f3394009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.431+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3390019100 con 0x7f33a40aee40 2024-03-31T15:53:59.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.587+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33a406f180 con 0x7f33a40aee40 2024-03-31T15:53:59.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.587+0000 7f339b400640 1 -- 172.21.15.90:0/2646652026 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f33900f43d0 con 0x7f33a40aee40 2024-03-31T15:53:59.582 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:53:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 msgr2=0x7f338406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 0x7f338406a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3394009be0 tx=0x7f3394009190 comp rx=0 tx=0).stop 2024-03-31T15:53:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 msgr2=0x7f33a4159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f339000c240 tx=0x7f3388005840 comp rx=0 tx=0).stop 2024-03-31T15:53:59.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 shutdown_connections 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3384068380 0x7f338406a840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f33a4159fd0 0x7f33a415a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33a40aee40 0x7f33a4159a90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 --2- 172.21.15.90:0/2646652026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3388003f70 0x7f33a40ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 >> 172.21.15.90:0/2646652026 conn(0x7f33a40d0390 msgr2=0x7f33a4074a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 shutdown_connections 2024-03-31T15:53:59.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.591+0000 7f33a3e00640 1 -- 172.21.15.90:0/2646652026 wait complete. 2024-03-31T15:53:59.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:53:59.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.747+0000 7f78ec200640 1 Processor -- start 2024-03-31T15:53:59.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.747+0000 7f78ec200640 1 -- start start 2024-03-31T15:53:59.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ec200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f78e406d020 0x7f78e40ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ec200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78e40cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ec200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78e415e8c0 con 0x7f78e415e4e0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ec200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f78e40d0ff0 con 0x7f78e406d020 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ec200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f78e40d12d0 con 0x7f78e406d490 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e9800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f78e415e4e0 0x7f78e406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57838/0 (socket says 172.21.15.90:57838) 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e9800640 1 -- 172.21.15.90:0/993035598 learned_addr learned my addr 172.21.15.90:0/993035598 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2231945963 0 0) 0x7f78e415e8c0 con 0x7f78e415e4e0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78cc003530 con 0x7f78e415e4e0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e8e00640 1 --2- 172.21.15.90:0/993035598 >> v2:172.21.15.102:3300/0 conn(0x7f78e406d020 0x7f78e40ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78e40cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2381249700 0 0) 0x7f78cc003530 con 0x7f78e415e4e0 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 msgr2=0x7f78e40cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78e40cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 >> v2:172.21.15.102:3300/0 conn(0x7f78e406d020 msgr2=0x7f78e40ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 --2- 172.21.15.90:0/993035598 >> v2:172.21.15.102:3300/0 conn(0x7f78e406d020 0x7f78e40ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e40cc710 con 0x7f78e415e4e0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f78d8001680 con 0x7f78e415e4e0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 >> v1:172.21.15.90:6789/0 conn(0x7f78e415e4e0 legacy=0x7f78e406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78cc0039c0 0x7f78cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78cc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78cc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78e40d0ff0 con 0x7f78cc004d50 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78e40d12d0 con 0x7f78cc0039c0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f78cc00ba00 con 0x7f78e406d490 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78cc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e8e00640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78cc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78e9800640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78cc0039c0 0x7f78cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 -- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 msgr2=0x7f78cc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e406d490 0x7f78cc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 -- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78cc0039c0 msgr2=0x7f78cc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:53:59.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78cc0039c0 0x7f78cc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.751+0000 7f78ea200640 1 -- 172.21.15.90:0/993035598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78cc003530 con 0x7f78cc004d50 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ea200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78cc0051d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009fc0 tx=0x7f78dc01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579937473994f19f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0b2070 con 0x7f78cc004d50 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78dc0a5b80 con 0x7f78cc004d50 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78e3e00640 1 -- 172.21.15.90:0/993035598 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc009c10 con 0x7f78cc004d50 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 -- 172.21.15.90:0/993035598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 msgr2=0x7f78cc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 --2- 172.21.15.90:0/993035598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78cc0051d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009fc0 tx=0x7f78dc01cc80 comp rx=0 tx=0).stop 2024-03-31T15:53:59.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ea200640 1 -- 172.21.15.90:0/993035598 reap_dead start 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 -- 172.21.15.90:0/993035598 shutdown_connections 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 -- 172.21.15.90:0/993035598 >> 172.21.15.90:0/993035598 conn(0x7f78e40d1f20 msgr2=0x7f78e40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 -- 172.21.15.90:0/993035598 shutdown_connections 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.755+0000 7f78ec200640 1 -- 172.21.15.90:0/993035598 wait complete. 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 Processor -- start 2024-03-31T15:53:59.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- start start 2024-03-31T15:53:59.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e415ba20 0x7f78e4153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78e40cb950 0x7f78e40cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78e40cf7b0 con 0x7f78cc004d50 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78e40d2d60 con 0x7f78e40cb950 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f78e40d0c10 con 0x7f78e415ba20 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43244/0 (socket says 172.21.15.90:43244) 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 -- 172.21.15.90:0/1430801676 learned_addr learned my addr 172.21.15.90:0/1430801676 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e8e00640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e415ba20 0x7f78e4153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ea200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78e40cb950 0x7f78e40cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 -- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e415ba20 msgr2=0x7f78e4153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e415ba20 0x7f78e4153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 -- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78e40cb950 msgr2=0x7f78e40cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78e40cb950 0x7f78e40cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 -- 172.21.15.90:0/1430801676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e40c9de0 con 0x7f78cc004d50 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e9800640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78d80124c0 tx=0x7f78d8012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d801c070 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78d8014810 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8014be0 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e40c9820 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.759+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e40cb150 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e40685f0 con 0x7f78cc004d50 2024-03-31T15:53:59.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f78d80153e0 con 0x7f78cc004d50 2024-03-31T15:53:59.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78e1600640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 0x7f78c406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:53:59.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f78d8089880 con 0x7f78cc004d50 2024-03-31T15:53:59.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78e8e00640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 0x7f78c406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:53:59.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.763+0000 7f78e8e00640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 0x7f78c406a870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78e40cb040 tx=0x7f78d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:53:59.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.767+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f78d8059a60 con 0x7f78cc004d50 2024-03-31T15:53:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78e40b2ea0 con 0x7f78cc004d50 2024-03-31T15:53:59.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78e1600640 1 -- 172.21.15.90:0/1430801676 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f78d8059360 con 0x7f78cc004d50 2024-03-31T15:53:59.925 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 msgr2=0x7f78c406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 0x7f78c406a870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f78e40cb040 tx=0x7f78d0008040 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 msgr2=0x7f78e40c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78d80124c0 tx=0x7f78d8012980 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 shutdown_connections 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78c40683b0 0x7f78c406a870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78e40cb950 0x7f78e40cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f78e415ba20 0x7f78e4153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 --2- 172.21.15.90:0/1430801676 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78cc004d50 0x7f78e40c8ff0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 >> 172.21.15.90:0/1430801676 conn(0x7f78e40d1f20 msgr2=0x7f78e40c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 shutdown_connections 2024-03-31T15:53:59.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:53:59.931+0000 7f78ec200640 1 -- 172.21.15.90:0/1430801676 wait complete. 2024-03-31T15:53:59.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.091+0000 7f7b7a200640 1 Processor -- start 2024-03-31T15:54:00.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.091+0000 7f7b7a200640 1 -- start start 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b7a200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b7406d490 0x7f7b740cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b7a200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7415e8c0 con 0x7f7b7406d020 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b7a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b740d0ff0 con 0x7f7b7415e4e0 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b7a200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7b740d12d0 con 0x7f7b7406d490 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7b7406d020 0x7f7b740ccf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57852/0 (socket says 172.21.15.90:57852) 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 -- 172.21.15.90:0/4035626279 learned_addr learned my addr 172.21.15.90:0/4035626279 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166087058 0 0) 0x7f7b7415e8c0 con 0x7f7b7406d020 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b58003530 con 0x7f7b7406d020 2024-03-31T15:54:00.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b73e00640 1 --2- 172.21.15.90:0/4035626279 >> v2:172.21.15.102:3300/0 conn(0x7f7b7406d490 0x7f7b740cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1605637455 0 0) 0x7f7b58003530 con 0x7f7b7406d020 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 >> v2:172.21.15.102:3300/0 conn(0x7f7b7406d490 msgr2=0x7f7b740cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 --2- 172.21.15.90:0/4035626279 >> v2:172.21.15.102:3300/0 conn(0x7f7b7406d490 0x7f7b740cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 >> v1:172.21.15.102:6789/0 conn(0x7f7b7415e4e0 legacy=0x7f7b7406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b740cc710 con 0x7f7b7406d020 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7b5c001680 con 0x7f7b7406d020 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 >> v1:172.21.15.90:6789/0 conn(0x7f7b7406d020 legacy=0x7f7b740ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b580039c0 0x7f7b58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b580052f0 0x7f7b5800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b740d12d0 con 0x7f7b58004810 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5800c1c0 con 0x7f7b580039c0 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b5800c3c0 con 0x7f7b580052f0 2024-03-31T15:54:00.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b73400640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b580039c0 0x7f7b58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b73e00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b580052f0 0x7f7b5800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 -- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b580052f0 msgr2=0x7f7b5800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b580052f0 0x7f7b5800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 -- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b580039c0 msgr2=0x7f7b58003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b580039c0 0x7f7b58003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b6be00640 1 -- 172.21.15.90:0/4035626279 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b58003530 con 0x7f7b58004810 2024-03-31T15:54:00.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.095+0000 7f7b73e00640 1 -- 172.21.15.90:0/4035626279 reap_dead start 2024-03-31T15:54:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b6be00640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 0x7f7b58004c90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c001ec0 tx=0x7f7b5c01cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea44abbf5bc0664b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c007d40 con 0x7f7b58004810 2024-03-31T15:54:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5c0a9070 con 0x7f7b58004810 2024-03-31T15:54:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b72a00640 1 -- 172.21.15.90:0/4035626279 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c005d80 con 0x7f7b58004810 2024-03-31T15:54:00.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 msgr2=0x7f7b58004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 0x7f7b58004c90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c001ec0 tx=0x7f7b5c01cbc0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- 172.21.15.90:0/4035626279 shutdown_connections 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 --2- 172.21.15.90:0/4035626279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- 172.21.15.90:0/4035626279 >> 172.21.15.90:0/4035626279 conn(0x7f7b740d1f20 msgr2=0x7f7b740d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- 172.21.15.90:0/4035626279 shutdown_connections 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- 172.21.15.90:0/4035626279 wait complete. 2024-03-31T15:54:00.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 Processor -- start 2024-03-31T15:54:00.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.099+0000 7f7b7a200640 1 -- start start 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b740cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b741600d0 0x7f7b74160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b740d0d90 con 0x7f7b7409f560 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b740d0c10 con 0x7f7b741600d0 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b740cefb0 con 0x7f7b58004810 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43252/0 (socket says 172.21.15.90:43252) 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 -- 172.21.15.90:0/1675288012 learned_addr learned my addr 172.21.15.90:0/1675288012 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b73400640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b740cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b73e00640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b741600d0 0x7f7b74160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 -- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 msgr2=0x7f7b740cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b740cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 -- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b741600d0 msgr2=0x7f7b74160550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b741600d0 0x7f7b74160550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 -- 172.21.15.90:0/1675288012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7415aa80 con 0x7f7b7409f560 2024-03-31T15:54:00.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6be00640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c01c630 tx=0x7f7b58005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c007d40 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5c000ee0 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c007180 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7415ad60 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b740cbc30 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.103+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b740685f0 con 0x7f7b7409f560 2024-03-31T15:54:00.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.107+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b5c007320 con 0x7f7b7409f560 2024-03-31T15:54:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.107+0000 7f7b6b400640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 0x7f7b5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.107+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7b5c1203b0 con 0x7f7b7409f560 2024-03-31T15:54:00.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.107+0000 7f7b73400640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 0x7f7b5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.107+0000 7f7b73400640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 0x7f7b5006a870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c10 tx=0x7f7b64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.111+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b5c0168d0 con 0x7f7b7409f560 2024-03-31T15:54:00.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7b7415e8c0 con 0x7f7b7409f560 2024-03-31T15:54:00.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b6b400640 1 -- 172.21.15.90:0/1675288012 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f7b5c0eff50 con 0x7f7b7409f560 2024-03-31T15:54:00.284 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 msgr2=0x7f7b5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 0x7f7b5006a870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000c10 tx=0x7f7b64009040 comp rx=0 tx=0).stop 2024-03-31T15:54:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 msgr2=0x7f7b7415a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c01c630 tx=0x7f7b58005840 comp rx=0 tx=0).stop 2024-03-31T15:54:00.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 shutdown_connections 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b500683b0 0x7f7b5006a870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b741600d0 0x7f7b74160550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7409f560 0x7f7b7415a130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 --2- 172.21.15.90:0/1675288012 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b740cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 >> 172.21.15.90:0/1675288012 conn(0x7f7b740d1f20 msgr2=0x7f7b740c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 shutdown_connections 2024-03-31T15:54:00.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.291+0000 7f7b7a200640 1 -- 172.21.15.90:0/1675288012 wait complete. 2024-03-31T15:54:00.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 Processor -- start 2024-03-31T15:54:00.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 -- start start 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e2815c1f0 0x7f4e280cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e280cc840 con 0x7f4e2815be10 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e280ccae0 con 0x7f4e280b43e0 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e2e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4e280cf740 con 0x7f4e2815c1f0 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e2815c1f0 0x7f4e280cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e2815c1f0 0x7f4e280cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43266/0 (socket says 172.21.15.90:43266) 2024-03-31T15:54:00.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 -- 172.21.15.90:0/3140910796 learned_addr learned my addr 172.21.15.90:0/3140910796 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 -- 172.21.15.90:0/3140910796 >> v1:172.21.15.102:6789/0 conn(0x7f4e280b43e0 legacy=0x7f4e280cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 -- 172.21.15.90:0/3140910796 >> v1:172.21.15.90:6789/0 conn(0x7f4e2815be10 legacy=0x7f4e280ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.451+0000 7f4e27e00640 1 -- 172.21.15.90:0/3140910796 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e280ccdc0 con 0x7f4e2815c1f0 2024-03-31T15:54:00.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e27e00640 1 --2- 172.21.15.90:0/3140910796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815c1f0 0x7f4e280cd8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eec2e3cebf5b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e27400640 1 -- 172.21.15.90:0/3140910796 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140ab660 con 0x7f4e2815c1f0 2024-03-31T15:54:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e27400640 1 -- 172.21.15.90:0/3140910796 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140b0070 con 0x7f4e2815c1f0 2024-03-31T15:54:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e27400640 1 -- 172.21.15.90:0/3140910796 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140aba20 con 0x7f4e2815c1f0 2024-03-31T15:54:00.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- 172.21.15.90:0/3140910796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815c1f0 msgr2=0x7f4e280cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 --2- 172.21.15.90:0/3140910796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815c1f0 0x7f4e280cd8a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009870 tx=0x7f4e1401c460 comp rx=0 tx=0).stop 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- 172.21.15.90:0/3140910796 shutdown_connections 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 --2- 172.21.15.90:0/3140910796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815c1f0 0x7f4e280cd8a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- 172.21.15.90:0/3140910796 >> 172.21.15.90:0/3140910796 conn(0x7f4e280b38c0 msgr2=0x7f4e280b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- 172.21.15.90:0/3140910796 shutdown_connections 2024-03-31T15:54:00.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- 172.21.15.90:0/3140910796 wait complete. 2024-03-31T15:54:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 Processor -- start 2024-03-31T15:54:00.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- start start 2024-03-31T15:54:00.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e280b43e0 0x7f4e2815efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e2815c1f0 0x7f4e2815c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2806fb30 con 0x7f4e2815be10 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2806fcb0 con 0x7f4e2815c1f0 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e2e200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4e2806fe30 con 0x7f4e280b43e0 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.459+0000 7f4e27e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43274/0 (socket says 172.21.15.90:43274) 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 -- 172.21.15.90:0/1424551425 learned_addr learned my addr 172.21.15.90:0/1424551425 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e2ce00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e280b43e0 0x7f4e2815efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e2d800640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e2815c1f0 0x7f4e2815c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 -- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e280b43e0 msgr2=0x7f4e2815efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e280b43e0 0x7f4e2815efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 -- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e2815c1f0 msgr2=0x7f4e2815c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e2815c1f0 0x7f4e2815c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 -- 172.21.15.90:0/1424551425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e280ca720 con 0x7f4e2815be10 2024-03-31T15:54:00.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e27e00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e140ad040 tx=0x7f4e1401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140b4800 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140b49a0 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2813d4f0 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140b4d90 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e280cac80 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e280685f0 con 0x7f4e2815be10 2024-03-31T15:54:00.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.463+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e140c7440 con 0x7f4e2815be10 2024-03-31T15:54:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.467+0000 7f4e24c00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 0x7f4e0806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.467+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4e14121650 con 0x7f4e2815be10 2024-03-31T15:54:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.467+0000 7f4e2ce00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 0x7f4e0806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.467+0000 7f4e2ce00640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 0x7f4e0806a870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c001650 tx=0x7f4e1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.467+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e280ca720 con 0x7f4e2815be10 2024-03-31T15:54:00.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.635+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e280d1b10 con 0x7f4e2815be10 2024-03-31T15:54:00.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.635+0000 7f4e24c00640 1 -- 172.21.15.90:0/1424551425 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f4e14016110 con 0x7f4e2815be10 2024-03-31T15:54:00.632 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 msgr2=0x7f4e0806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 0x7f4e0806a870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c001650 tx=0x7f4e1c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 msgr2=0x7f4e280c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e140ad040 tx=0x7f4e1401aca0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 shutdown_connections 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e080683b0 0x7f4e0806a870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e2815c1f0 0x7f4e2815c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e2815be10 0x7f4e280c8e00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 --2- 172.21.15.90:0/1424551425 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e280b43e0 0x7f4e2815efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 >> 172.21.15.90:0/1424551425 conn(0x7f4e280b38c0 msgr2=0x7f4e280b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:00.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 shutdown_connections 2024-03-31T15:54:00.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.639+0000 7f4e2e200640 1 -- 172.21.15.90:0/1424551425 wait complete. 2024-03-31T15:54:00.652 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:00.652 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:00.660 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:00.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 Processor -- start 2024-03-31T15:54:00.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 -- start start 2024-03-31T15:54:00.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f31a415e1b0 0x7f31a4162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f31a406c7b0 0x7f31a406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a406d5f0 0x7f31a415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f31a41622b0 con 0x7f31a415e1b0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f31a406da60 con 0x7f31a406c7b0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31ac200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f31a415f780 con 0x7f31a406d5f0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f31a415e1b0 0x7f31a4162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f31a415e1b0 0x7f31a4162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43288/0 (socket says 172.21.15.90:43288) 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 -- 172.21.15.90:0/3869767768 learned_addr learned my addr 172.21.15.90:0/3869767768 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31aa200640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a406d5f0 0x7f31a415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a8e00640 1 --2- 172.21.15.90:0/3869767768 >> v2:172.21.15.102:3300/0 conn(0x7f31a406c7b0 0x7f31a406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 -- 172.21.15.90:0/3869767768 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a406d5f0 msgr2=0x7f31a415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a406d5f0 0x7f31a415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 -- 172.21.15.90:0/3869767768 >> v2:172.21.15.102:3300/0 conn(0x7f31a406c7b0 msgr2=0x7f31a406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:00.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 --2- 172.21.15.90:0/3869767768 >> v2:172.21.15.102:3300/0 conn(0x7f31a406c7b0 0x7f31a406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.859+0000 7f31a9800640 1 -- 172.21.15.90:0/3869767768 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a415f1c0 con 0x7f31a415e1b0 2024-03-31T15:54:00.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31a9800640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a415e1b0 0x7f31a4162740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31a406ed50 tx=0x7f319801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b25ab195f7b3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31a3e00640 1 -- 172.21.15.90:0/3869767768 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31980ab620 con 0x7f31a415e1b0 2024-03-31T15:54:00.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31a3e00640 1 -- 172.21.15.90:0/3869767768 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31980b0070 con 0x7f31a415e1b0 2024-03-31T15:54:00.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31a3e00640 1 -- 172.21.15.90:0/3869767768 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31980ab9e0 con 0x7f31a415e1b0 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 -- 172.21.15.90:0/3869767768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a415e1b0 msgr2=0x7f31a4162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a415e1b0 0x7f31a4162740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31a406ed50 tx=0x7f319801c460 comp rx=0 tx=0).stop 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 -- 172.21.15.90:0/3869767768 shutdown_connections 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a406d5f0 0x7f31a415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 --2- 172.21.15.90:0/3869767768 >> v2:172.21.15.102:3300/0 conn(0x7f31a406c7b0 0x7f31a406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 --2- 172.21.15.90:0/3869767768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a415e1b0 0x7f31a4162740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 -- 172.21.15.90:0/3869767768 >> 172.21.15.90:0/3869767768 conn(0x7f31a409aa40 msgr2=0x7f31a409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 -- 172.21.15.90:0/3869767768 shutdown_connections 2024-03-31T15:54:00.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.863+0000 7f31ac200640 1 -- 172.21.15.90:0/3869767768 wait complete. 2024-03-31T15:54:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 Processor -- start 2024-03-31T15:54:00.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- start start 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a406c7b0 0x7f31a40c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a415e1b0 0x7f31a41553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31a40d27c0 con 0x7f31a406d5f0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4162c80 con 0x7f31a406c7b0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f31a40d2940 con 0x7f31a415e1b0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43300/0 (socket says 172.21.15.90:43300) 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 -- 172.21.15.90:0/2345762378 learned_addr learned my addr 172.21.15.90:0/2345762378 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a9800640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a406c7b0 0x7f31a40c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31aa200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a415e1b0 0x7f31a41553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 -- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a415e1b0 msgr2=0x7f31a41553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a415e1b0 0x7f31a41553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 -- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a406c7b0 msgr2=0x7f31a40c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a406c7b0 0x7f31a40c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 -- 172.21.15.90:0/2345762378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a40cc0c0 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a8e00640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f319000d5f0 tx=0x7f319000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31900125f0 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190019070 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a40cb820 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31900129b0 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a40cbde0 con 0x7f31a406d5f0 2024-03-31T15:54:00.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.867+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a40686d0 con 0x7f31a406d5f0 2024-03-31T15:54:00.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.871+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3190012b50 con 0x7f31a406d5f0 2024-03-31T15:54:00.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.871+0000 7f31a1600640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 0x7f318406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:00.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.871+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3190086f10 con 0x7f31a406d5f0 2024-03-31T15:54:00.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.871+0000 7f31a9800640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 0x7f318406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:00.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.871+0000 7f31a9800640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 0x7f318406a870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31980af000 tx=0x7f31980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:00.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:00.875+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3190056bd0 con 0x7f31a406d5f0 2024-03-31T15:54:01.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31a40b1b60 con 0x7f31a406d5f0 2024-03-31T15:54:01.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31a1600640 1 -- 172.21.15.90:0/2345762378 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f319005aed0 con 0x7f31a406d5f0 2024-03-31T15:54:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:01.029 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:01.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 msgr2=0x7f318406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 0x7f318406a870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f31980af000 tx=0x7f31980a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 msgr2=0x7f31a40c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.035+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f319000d5f0 tx=0x7f319000dab0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 shutdown_connections 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31840683b0 0x7f318406a870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31a415e1b0 0x7f31a41553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31a406d5f0 0x7f31a40c8410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 --2- 172.21.15.90:0/2345762378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a406c7b0 0x7f31a40c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 >> 172.21.15.90:0/2345762378 conn(0x7f31a409aa40 msgr2=0x7f31a40cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 shutdown_connections 2024-03-31T15:54:01.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.039+0000 7f31ac200640 1 -- 172.21.15.90:0/2345762378 wait complete. 2024-03-31T15:54:01.050 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:01.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:01.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.195+0000 7fd8c1200640 1 Processor -- start 2024-03-31T15:54:01.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.195+0000 7fd8c1200640 1 -- start start 2024-03-31T15:54:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8c1200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 0x7fd8bc0afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8c1200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8bc153170 0x7fd8bc0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8c1200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8bc15d550 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8c1200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd8bc0cea60 con 0x7fd8bc153170 2024-03-31T15:54:01.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8c1200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd8bc0cd6d0 con 0x7fd8bc0d03c0 2024-03-31T15:54:01.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8bc153170 0x7fd8bc0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8b9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 0x7fd8bc0afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd8bc153170 0x7fd8bc0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43302/0 (socket says 172.21.15.90:43302) 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 -- 172.21.15.90:0/2729607546 learned_addr learned my addr 172.21.15.90:0/2729607546 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 -- 172.21.15.90:0/2729607546 >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 msgr2=0x7fd8bc0afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 --2- 172.21.15.90:0/2729607546 >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 0x7fd8bc0afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 -- 172.21.15.90:0/2729607546 >> v1:172.21.15.90:6789/0 conn(0x7fd8bc0d07a0 legacy=0x7fd8bc152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8ba800640 1 -- 172.21.15.90:0/2729607546 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc15c790 con 0x7fd8bc153170 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8b8a00640 1 -- 172.21.15.90:0/2729607546 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2318151838 0 0) 0x7fd8bc15d550 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.199+0000 7fd8b9e00640 1 --2- 172.21.15.90:0/2729607546 >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 0x7fd8bc0afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8ba800640 1 --2- 172.21.15.90:0/2729607546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc153170 0x7fd8bc0b02e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0cd870 tx=0x7fd8b401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a74aed47b4e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8b8a00640 1 -- 172.21.15.90:0/2729607546 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40a56c0 con 0x7fd8bc153170 2024-03-31T15:54:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8b8a00640 1 -- 172.21.15.90:0/2729607546 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b40a5860 con 0x7fd8bc153170 2024-03-31T15:54:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8b8a00640 1 -- 172.21.15.90:0/2729607546 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40a5c30 con 0x7fd8bc153170 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- 172.21.15.90:0/2729607546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc153170 msgr2=0x7fd8bc0b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 --2- 172.21.15.90:0/2729607546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc153170 0x7fd8bc0b02e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0cd870 tx=0x7fd8b401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- 172.21.15.90:0/2729607546 shutdown_connections 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 --2- 172.21.15.90:0/2729607546 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc153170 0x7fd8bc0b02e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 --2- 172.21.15.90:0/2729607546 >> v2:172.21.15.102:3300/0 conn(0x7fd8bc0d03c0 0x7fd8bc0afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- 172.21.15.90:0/2729607546 >> 172.21.15.90:0/2729607546 conn(0x7fd8bc0b29a0 msgr2=0x7fd8bc0b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- 172.21.15.90:0/2729607546 shutdown_connections 2024-03-31T15:54:01.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- 172.21.15.90:0/2729607546 wait complete. 2024-03-31T15:54:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 Processor -- start 2024-03-31T15:54:01.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.203+0000 7fd8c1200640 1 -- start start 2024-03-31T15:54:01.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8bc0d03c0 0x7fd8bc06fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8bc153170 0x7fd8bc153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc06bff0 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc06be70 con 0x7fd8bc0d03c0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd8bc06c170 con 0x7fd8bc153170 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43304/0 (socket says 172.21.15.90:43304) 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 -- 172.21.15.90:0/3613891733 learned_addr learned my addr 172.21.15.90:0/3613891733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9e00640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8bc0d03c0 0x7fd8bc06fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8ba800640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8bc153170 0x7fd8bc153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 -- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8bc153170 msgr2=0x7fd8bc153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8bc153170 0x7fd8bc153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 -- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8bc0d03c0 msgr2=0x7fd8bc06fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8bc0d03c0 0x7fd8bc06fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 -- 172.21.15.90:0/3613891733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc152160 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b9400640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c4d0 tx=0x7fd8a800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8017070 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8a8005480 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a8003840 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8bc0c6670 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8bc0c6c30 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8bc0685f0 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.207+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8a801a070 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.211+0000 7fd8b2000640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 0x7fd89c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.211+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd8a8086110 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.211+0000 7fd8b9e00640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 0x7fd89c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.211+0000 7fd8b9e00640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 0x7fd89c06a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009580 tx=0x7fd8ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.219+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd8a8055d60 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.375+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8bc072a80 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.375+0000 7fd8b2000640 1 -- 172.21.15.90:0/3613891733 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fd8a805a060 con 0x7fd8bc0d07a0 2024-03-31T15:54:01.370 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 msgr2=0x7fd89c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 0x7fd89c06a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009580 tx=0x7fd8ac009190 comp rx=0 tx=0).stop 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 msgr2=0x7fd8bc151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800c4d0 tx=0x7fd8a800c990 comp rx=0 tx=0).stop 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 shutdown_connections 2024-03-31T15:54:01.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd89c068380 0x7fd89c06a840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8bc153170 0x7fd8bc153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc0d07a0 0x7fd8bc151c20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 --2- 172.21.15.90:0/3613891733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8bc0d03c0 0x7fd8bc06fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 >> 172.21.15.90:0/3613891733 conn(0x7fd8bc0b29a0 msgr2=0x7fd8bc0cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 shutdown_connections 2024-03-31T15:54:01.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.379+0000 7fd8c1200640 1 -- 172.21.15.90:0/3613891733 wait complete. 2024-03-31T15:54:01.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:01.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.535+0000 7f786b200640 1 Processor -- start 2024-03-31T15:54:01.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.535+0000 7f786b200640 1 -- start start 2024-03-31T15:54:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f786c1475f0 0x7f786c0714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c0719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f786c1479d0 con 0x7f786c0b3010 2024-03-31T15:54:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f786c0ae780 con 0x7f786c1475f0 2024-03-31T15:54:01.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f786c0aea60 con 0x7f786c0ae2c0 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f786c1475f0 0x7f786c0714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f786c0b3010 0x7f786c0b34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57868/0 (socket says 172.21.15.90:57868) 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f786a800640 1 -- 172.21.15.90:0/3039228305 learned_addr learned my addr 172.21.15.90:0/3039228305 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7863e00640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c0719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869e00640 1 -- 172.21.15.90:0/3039228305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 msgr2=0x7f786c0719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:01.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869e00640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c0719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869e00640 1 -- 172.21.15.90:0/3039228305 >> v1:172.21.15.90:6789/0 conn(0x7f786c0b3010 legacy=0x7f786c0b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869e00640 1 -- 172.21.15.90:0/3039228305 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c0aed40 con 0x7f786c1475f0 2024-03-31T15:54:01.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.539+0000 7f7869400640 1 -- 172.21.15.90:0/3039228305 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723530589 0 0) 0x7f786c1479d0 con 0x7f786c0b3010 2024-03-31T15:54:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f7869e00640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c1475f0 0x7f786c0714b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f785c0097c0 tx=0x7f785c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47755c1b360fc0b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f7869400640 1 -- 172.21.15.90:0/3039228305 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0b1070 con 0x7f786c1475f0 2024-03-31T15:54:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f7869400640 1 -- 172.21.15.90:0/3039228305 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c0ac640 con 0x7f786c1475f0 2024-03-31T15:54:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f7869400640 1 -- 172.21.15.90:0/3039228305 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0aca10 con 0x7f786c1475f0 2024-03-31T15:54:01.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 -- 172.21.15.90:0/3039228305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c1475f0 msgr2=0x7f786c0714b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c1475f0 0x7f786c0714b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f785c0097c0 tx=0x7f785c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 -- 172.21.15.90:0/3039228305 shutdown_connections 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c0719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 --2- 172.21.15.90:0/3039228305 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c1475f0 0x7f786c0714b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 -- 172.21.15.90:0/3039228305 >> 172.21.15.90:0/3039228305 conn(0x7f786c0d0040 msgr2=0x7f786c0d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 -- 172.21.15.90:0/3039228305 shutdown_connections 2024-03-31T15:54:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.543+0000 7f786b200640 1 -- 172.21.15.90:0/3039228305 wait complete. 2024-03-31T15:54:01.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 Processor -- start 2024-03-31T15:54:01.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 -- start start 2024-03-31T15:54:01.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c13ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 0x7f786c0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f786c16aeb0 con 0x7f786c0b3010 2024-03-31T15:54:01.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f786c06fef0 con 0x7f786c1475f0 2024-03-31T15:54:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f786b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f786c16b030 con 0x7f786c0ae2c0 2024-03-31T15:54:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f7863e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f7863e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43322/0 (socket says 172.21.15.90:43322) 2024-03-31T15:54:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.547+0000 7f7863e00640 1 -- 172.21.15.90:0/766595291 learned_addr learned my addr 172.21.15.90:0/766595291 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:01.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7869e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c13ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f786a800640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 0x7f786c0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 -- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 msgr2=0x7f786c13ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c13ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 -- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 msgr2=0x7f786c0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 0x7f786c0b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 -- 172.21.15.90:0/766595291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c13b360 con 0x7f786c0b3010 2024-03-31T15:54:01.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f786a800640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 0x7f786c0b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7863e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f785400d800 tx=0x7f785400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785400b6c0 con 0x7f786c0b3010 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7854019070 con 0x7f786c0b3010 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785400ba60 con 0x7f786c0b3010 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c0cc220 con 0x7f786c0b3010 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1fb2c0 con 0x7f786c0b3010 2024-03-31T15:54:01.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c068d00 con 0x7f786c0b3010 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f785400bc00 con 0x7f786c0b3010 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 0x7f784806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.551+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7854087140 con 0x7f786c0b3010 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.555+0000 7f7869e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 0x7f784806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.555+0000 7f7869e00640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 0x7f784806a840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f785c0ae000 tx=0x7f785c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.555+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f78540573b0 con 0x7f786c0b3010 2024-03-31T15:54:01.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.735+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f786c13dcb0 con 0x7f786c0b3010 2024-03-31T15:54:01.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.735+0000 7f7862000640 1 -- 172.21.15.90:0/766595291 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f7854056cb0 con 0x7f786c0b3010 2024-03-31T15:54:01.731 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:01.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 msgr2=0x7f784806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 0x7f784806a840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f785c0ae000 tx=0x7f785c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 msgr2=0x7f786c140f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f785400d800 tx=0x7f785400dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 shutdown_connections 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7848068380 0x7f784806a840 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f786c1475f0 0x7f786c0b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f786c0b3010 0x7f786c140f00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 --2- 172.21.15.90:0/766595291 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f786c0ae2c0 0x7f786c13ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 >> 172.21.15.90:0/766595291 conn(0x7f786c0d0040 msgr2=0x7f786c0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 shutdown_connections 2024-03-31T15:54:01.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.739+0000 7f786b200640 1 -- 172.21.15.90:0/766595291 wait complete. 2024-03-31T15:54:01.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.895+0000 7fe730000640 1 Processor -- start 2024-03-31T15:54:01.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.895+0000 7fe730000640 1 -- start start 2024-03-31T15:54:01.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe730000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe72815c320 0x7fe7280cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe730000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7280d2b20 0x7fe7280ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe730000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7280cd900 con 0x7fe7280cd470 2024-03-31T15:54:01.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe730000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7280aed70 con 0x7fe72815c320 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe730000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7280cefc0 con 0x7fe7280d2b20 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72e000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe7280cd470 0x7fe7280cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57876/0 (socket says 172.21.15.90:57876) 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72e000640 1 -- 172.21.15.90:0/1164122175 learned_addr learned my addr 172.21.15.90:0/1164122175 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72d600640 1 --2- 172.21.15.90:0/1164122175 >> v2:172.21.15.102:3300/0 conn(0x7fe72815c320 0x7fe7280cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72cc00640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7280d2b20 0x7fe7280ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe727e00640 1 -- 172.21.15.90:0/1164122175 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282017178 0 0) 0x7fe7280cd900 con 0x7fe7280cd470 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe727e00640 1 -- 172.21.15.90:0/1164122175 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe710003530 con 0x7fe7280cd470 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72d600640 1 -- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7280d2b20 msgr2=0x7fe7280ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72d600640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7280d2b20 0x7fe7280ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72d600640 1 -- 172.21.15.90:0/1164122175 >> v1:172.21.15.90:6789/0 conn(0x7fe7280cd470 legacy=0x7fe7280cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.899+0000 7fe72d600640 1 -- 172.21.15.90:0/1164122175 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7280cf2a0 con 0x7fe72815c320 2024-03-31T15:54:01.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe72d600640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe72815c320 0x7fe7280cc9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0097c0 tx=0x7fe71c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2804ecba16b9d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe727e00640 1 -- 172.21.15.90:0/1164122175 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0b0070 con 0x7fe72815c320 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe727e00640 1 -- 172.21.15.90:0/1164122175 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71c0ab600 con 0x7fe72815c320 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe727e00640 1 -- 172.21.15.90:0/1164122175 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0ab9b0 con 0x7fe72815c320 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 -- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe72815c320 msgr2=0x7fe7280cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe72815c320 0x7fe7280cc9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0097c0 tx=0x7fe71c01c410 comp rx=0 tx=0).stop 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 -- 172.21.15.90:0/1164122175 shutdown_connections 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7280d2b20 0x7fe7280ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 --2- 172.21.15.90:0/1164122175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe72815c320 0x7fe7280cc9f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 -- 172.21.15.90:0/1164122175 >> 172.21.15.90:0/1164122175 conn(0x7fe7280d1f20 msgr2=0x7fe7280d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:01.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 -- 172.21.15.90:0/1164122175 shutdown_connections 2024-03-31T15:54:01.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.903+0000 7fe730000640 1 -- 172.21.15.90:0/1164122175 wait complete. 2024-03-31T15:54:01.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 Processor -- start 2024-03-31T15:54:01.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 -- start start 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe72815c320 0x7fe728070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe728168ec0 con 0x7fe7280d2b20 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7280d0eb0 con 0x7fe7280cd470 2024-03-31T15:54:01.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe728169040 con 0x7fe72815c320 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43324/0 (socket says 172.21.15.90:43324) 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:39960/0 (socket says 172.21.15.90:39960) 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72e000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe72815c320 0x7fe728070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 -- 172.21.15.90:0/2775383534 learned_addr learned my addr 172.21.15.90:0/2775383534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 -- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe72815c320 msgr2=0x7fe728070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe72815c320 0x7fe728070590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 -- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 msgr2=0x7fe72814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 -- 172.21.15.90:0/2775383534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7280b1510 con 0x7fe7280d2b20 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72d600640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe72cc00640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe71400e8a0 tx=0x7fe71400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714017070 con 0x7fe7280d2b20 2024-03-31T15:54:01.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe71400c940 con 0x7fe7280d2b20 2024-03-31T15:54:01.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71400ccf0 con 0x7fe7280d2b20 2024-03-31T15:54:01.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.907+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7280b17f0 con 0x7fe7280d2b20 2024-03-31T15:54:01.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7280b1db0 con 0x7fe7280d2b20 2024-03-31T15:54:01.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7280685f0 con 0x7fe7280d2b20 2024-03-31T15:54:01.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe71401a070 con 0x7fe7280d2b20 2024-03-31T15:54:01.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe725600640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 0x7fe70806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:01.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe714086b40 con 0x7fe7280d2b20 2024-03-31T15:54:01.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe72d600640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 0x7fe70806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:01.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.911+0000 7fe72d600640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 0x7fe70806a840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0ad000 tx=0x7fe71c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:01.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:01.915+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe714056760 con 0x7fe7280d2b20 2024-03-31T15:54:02.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe72815ccf0 con 0x7fe7280d2b20 2024-03-31T15:54:02.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe725600640 1 -- 172.21.15.90:0/2775383534 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe71405aa60 con 0x7fe7280d2b20 2024-03-31T15:54:02.072 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 msgr2=0x7fe70806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 0x7fe70806a840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe71c0ad000 tx=0x7fe71c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 msgr2=0x7fe72814ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe71400e8a0 tx=0x7fe71400eca0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 shutdown_connections 2024-03-31T15:54:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe708068380 0x7fe70806a840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe72815c320 0x7fe728070590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7280d2b20 0x7fe72814ae30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 --2- 172.21.15.90:0/2775383534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7280cd470 0x7fe72814a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 >> 172.21.15.90:0/2775383534 conn(0x7fe7280d1f20 msgr2=0x7fe72815da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:02.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 shutdown_connections 2024-03-31T15:54:02.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.079+0000 7fe730000640 1 -- 172.21.15.90:0/2775383534 wait complete. 2024-03-31T15:54:02.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:02.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 Processor -- start 2024-03-31T15:54:02.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 -- start start 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe63415e4e0 0x7fe63406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63406ce50 0x7fe63406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 0x7fe6340cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe63415e8c0 con 0x7fe63415e4e0 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe6340d0ff0 con 0x7fe6340ce7a0 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe63b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe6340d12d0 con 0x7fe63406ce50 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe63415e4e0 0x7fe63406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe63415e4e0 0x7fe63406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43334/0 (socket says 172.21.15.90:43334) 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 -- 172.21.15.90:0/3305314910 learned_addr learned my addr 172.21.15.90:0/3305314910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe639400640 1 --2- 172.21.15.90:0/3305314910 >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 0x7fe6340cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe633e00640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63406ce50 0x7fe63406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 -- 172.21.15.90:0/3305314910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63406ce50 msgr2=0x7fe63406d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63406ce50 0x7fe63406d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 -- 172.21.15.90:0/3305314910 >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 msgr2=0x7fe6340cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 --2- 172.21.15.90:0/3305314910 >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 0x7fe6340cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe638a00640 1 -- 172.21.15.90:0/3305314910 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6340cc710 con 0x7fe63415e4e0 2024-03-31T15:54:02.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.259+0000 7fe639400640 1 --2- 172.21.15.90:0/3305314910 >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 0x7fe6340cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.263+0000 7fe638a00640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63415e4e0 0x7fe63406c910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe634075360 tx=0x7fe62801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8097bddad20706c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.263+0000 7fe633400640 1 -- 172.21.15.90:0/3305314910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6280b0070 con 0x7fe63415e4e0 2024-03-31T15:54:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.263+0000 7fe633400640 1 -- 172.21.15.90:0/3305314910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6280ab600 con 0x7fe63415e4e0 2024-03-31T15:54:02.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.263+0000 7fe633400640 1 -- 172.21.15.90:0/3305314910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6280ab9d0 con 0x7fe63415e4e0 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- 172.21.15.90:0/3305314910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63415e4e0 msgr2=0x7fe63406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63415e4e0 0x7fe63406c910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe634075360 tx=0x7fe62801c490 comp rx=0 tx=0).stop 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- 172.21.15.90:0/3305314910 shutdown_connections 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- 172.21.15.90:0/3305314910 >> v2:172.21.15.102:3300/0 conn(0x7fe6340ce7a0 0x7fe6340cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63406ce50 0x7fe63406d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- 172.21.15.90:0/3305314910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63415e4e0 0x7fe63406c910 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- 172.21.15.90:0/3305314910 >> 172.21.15.90:0/3305314910 conn(0x7fe6340d1f20 msgr2=0x7fe6340d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- 172.21.15.90:0/3305314910 shutdown_connections 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- 172.21.15.90:0/3305314910 wait complete. 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 Processor -- start 2024-03-31T15:54:02.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- start start 2024-03-31T15:54:02.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6340ce7a0 0x7fe634155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 0x7fe6341fad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6340cd090 con 0x7fe63406ce50 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6340ccf10 con 0x7fe6340ce7a0 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe63b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe6340cd210 con 0x7fe63415e4e0 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43350/0 (socket says 172.21.15.90:43350) 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 -- 172.21.15.90:0/3792650277 learned_addr learned my addr 172.21.15.90:0/3792650277 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe633e00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6340ce7a0 0x7fe634155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe639400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 0x7fe6341fad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 -- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 msgr2=0x7fe6341fad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 0x7fe6341fad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 -- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6340ce7a0 msgr2=0x7fe634155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6340ce7a0 0x7fe634155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 -- 172.21.15.90:0/3792650277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6341fb770 con 0x7fe63406ce50 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe639400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 0x7fe6341fad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.267+0000 7fe638a00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6280ae040 tx=0x7fe62801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6280b0040 con 0x7fe63406ce50 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6280bc070 con 0x7fe63406ce50 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6280b6880 con 0x7fe63406ce50 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6341fba50 con 0x7fe63406ce50 2024-03-31T15:54:02.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe634154d50 con 0x7fe63406ce50 2024-03-31T15:54:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6340685f0 con 0x7fe63406ce50 2024-03-31T15:54:02.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe6280ab480 con 0x7fe63406ce50 2024-03-31T15:54:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 0x7fe61406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe628120f70 con 0x7fe63406ce50 2024-03-31T15:54:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.271+0000 7fe633e00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 0x7fe61406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.275+0000 7fe633e00640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 0x7fe61406a840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe63415d690 tx=0x7fe620008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.275+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe628016110 con 0x7fe63406ce50 2024-03-31T15:54:02.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe634154840 con 0x7fe63406ce50 2024-03-31T15:54:02.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe630c00640 1 -- 172.21.15.90:0/3792650277 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fe6280f13a0 con 0x7fe63406ce50 2024-03-31T15:54:02.429 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 msgr2=0x7fe61406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 0x7fe61406a840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe63415d690 tx=0x7fe620008040 comp rx=0 tx=0).stop 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 msgr2=0x7fe634153f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6280ae040 tx=0x7fe62801acd0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 shutdown_connections 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe614068380 0x7fe61406a840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe63415e4e0 0x7fe6341fad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6340ce7a0 0x7fe634155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 --2- 172.21.15.90:0/3792650277 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe63406ce50 0x7fe634153f80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 >> 172.21.15.90:0/3792650277 conn(0x7fe6340d1f20 msgr2=0x7fe6340c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 shutdown_connections 2024-03-31T15:54:02.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.435+0000 7fe63b400640 1 -- 172.21.15.90:0/3792650277 wait complete. 2024-03-31T15:54:02.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:02.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.591+0000 7f5113a00640 1 Processor -- start 2024-03-31T15:54:02.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.591+0000 7f5113a00640 1 -- start start 2024-03-31T15:54:02.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.591+0000 7f5113a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f510c0cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5113a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f510c0d0ea0 con 0x7f510c15d240 2024-03-31T15:54:02.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5113a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f510c0cf230 con 0x7f510c163bd0 2024-03-31T15:54:02.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5113a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f510c0cd220 con 0x7f510c0ce910 2024-03-31T15:54:02.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f510c15d240 0x7f510c0ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57886/0 (socket says 172.21.15.90:57886) 2024-03-31T15:54:02.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510be00640 1 -- 172.21.15.90:0/2453730921 learned_addr learned my addr 172.21.15.90:0/2453730921 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:02.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111000640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f510c0cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322255349 0 0) 0x7f510c0d0ea0 con 0x7f510c15d240 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f4003530 con 0x7f510c15d240 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2604747301 0 0) 0x7f50f4003530 con 0x7f510c15d240 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 msgr2=0x7f510c0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f510c0cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 >> v1:172.21.15.102:6789/0 conn(0x7f510c163bd0 legacy=0x7f510c165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c0cd460 con 0x7f510c15d240 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f50f8001680 con 0x7f510c15d240 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 >> v1:172.21.15.90:6789/0 conn(0x7f510c15d240 legacy=0x7f510c0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f40039f0 0x7f50f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f50f4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f4004d50 0x7f50f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f510c0cd220 con 0x7f50f4004d50 2024-03-31T15:54:02.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f50f400ba80 con 0x7f50f40039f0 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f50f400bc80 con 0x7f510c0ce910 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f4004d50 0x7f50f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111000640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f40039f0 0x7f50f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f510be00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f50f4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 -- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 msgr2=0x7f50f4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c0ce910 0x7f50f4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 -- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f40039f0 msgr2=0x7f50f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f40039f0 0x7f50f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111a00640 1 -- 172.21.15.90:0/2453730921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f4003530 con 0x7f50f4004d50 2024-03-31T15:54:02.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.595+0000 7f5111000640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f40039f0 0x7f50f4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:02.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5111a00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f4004d50 0x7f50f40051d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f510c0cc600 tx=0x7f510401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbad16d652c4669 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104001b60 con 0x7f50f4004d50 2024-03-31T15:54:02.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51040a66f0 con 0x7f50f4004d50 2024-03-31T15:54:02.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f510b400640 1 -- 172.21.15.90:0/2453730921 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040a6ae0 con 0x7f50f4004d50 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 -- 172.21.15.90:0/2453730921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f4004d50 msgr2=0x7f50f40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 --2- 172.21.15.90:0/2453730921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f50f4004d50 0x7f50f40051d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f510c0cc600 tx=0x7f510401c7f0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5111a00640 1 -- 172.21.15.90:0/2453730921 reap_dead start 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 -- 172.21.15.90:0/2453730921 shutdown_connections 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 -- 172.21.15.90:0/2453730921 >> 172.21.15.90:0/2453730921 conn(0x7f510c08bfd0 msgr2=0x7f510c0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 -- 172.21.15.90:0/2453730921 shutdown_connections 2024-03-31T15:54:02.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.599+0000 7f5113a00640 1 -- 172.21.15.90:0/2453730921 wait complete. 2024-03-31T15:54:02.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 Processor -- start 2024-03-31T15:54:02.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- start start 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f4004d50 0x7f510c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 0x7f510c15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f510c167ef0 con 0x7f510c06e4a0 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f510c15d780 con 0x7f50f4004d50 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f510c168070 con 0x7f510c15eef0 2024-03-31T15:54:02.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43378/0 (socket says 172.21.15.90:43378) 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 -- 172.21.15.90:0/3979036733 learned_addr learned my addr 172.21.15.90:0/3979036733 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5111000640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f4004d50 0x7f510c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5111a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 0x7f510c15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 -- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 msgr2=0x7f510c15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 0x7f510c15f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 -- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f4004d50 msgr2=0x7f510c14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f4004d50 0x7f510c14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 -- 172.21.15.90:0/3979036733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c0c81f0 con 0x7f510c06e4a0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5111a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 0x7f510c15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f510be00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50f80072f0 tx=0x7f50f80076f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f800f9c0 con 0x7f510c06e4a0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f8018070 con 0x7f510c06e4a0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f800fdc0 con 0x7f510c06e4a0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510c0c84d0 con 0x7f510c06e4a0 2024-03-31T15:54:02.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.603+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510c0c7e30 con 0x7f510c06e4a0 2024-03-31T15:54:02.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f510c15f8b0 con 0x7f510c06e4a0 2024-03-31T15:54:02.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f50f801c410 con 0x7f510c06e4a0 2024-03-31T15:54:02.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5108c00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 0x7f50ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:02.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f50f8086820 con 0x7f510c06e4a0 2024-03-31T15:54:02.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5111000640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 0x7f50ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:02.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.607+0000 7f5111000640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 0x7f50ec06a870 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5100009d80 tx=0x7f5100009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:02.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.611+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f50f8056b30 con 0x7f510c06e4a0 2024-03-31T15:54:02.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f510c068660 con 0x7f510c06e4a0 2024-03-31T15:54:02.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5108c00640 1 -- 172.21.15.90:0/3979036733 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f50f8056430 con 0x7f510c06e4a0 2024-03-31T15:54:02.797 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 msgr2=0x7f50ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 0x7f50ec06a870 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5100009d80 tx=0x7f5100009190 comp rx=0 tx=0).stop 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 msgr2=0x7f510c0c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50f80072f0 tx=0x7f50f80076f0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 shutdown_connections 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f50ec0683b0 0x7f50ec06a870 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f510c15eef0 0x7f510c15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f510c06e4a0 0x7f510c0c78f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 --2- 172.21.15.90:0/3979036733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50f4004d50 0x7f510c14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:02.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 >> 172.21.15.90:0/3979036733 conn(0x7f510c08bfd0 msgr2=0x7f510c15a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:02.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 shutdown_connections 2024-03-31T15:54:02.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:02.803+0000 7f5113a00640 1 -- 172.21.15.90:0/3979036733 wait complete. 2024-03-31T15:54:02.820 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:54:02.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:02.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-31T15:54:02.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi090.stderr:ceph-fuse[33371]: fuse finished with error 0 and tester_r 0 2024-03-31T15:54:02.847 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:54:08.949 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:54:08.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:08.949 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-31T15:54:08.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 Processor -- start 2024-03-31T15:54:09.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 -- start start 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b6015c120 0x7f6b6015fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b600d2680 0x7f6b600b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b600b2c20 0x7f6b600cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6b60160bf0 con 0x7f6b6015c120 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6b60160350 con 0x7f6b600b2c20 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b65600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b60160630 con 0x7f6b600d2680 2024-03-31T15:54:09.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b5ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b6015c120 0x7f6b6015fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b5ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b6015c120 0x7f6b6015fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51032/0 (socket says 172.21.15.90:51032) 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.150+0000 7f6b5ea00640 1 -- 172.21.15.90:0/2615165856 learned_addr learned my addr 172.21.15.90:0/2615165856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 -- 172.21.15.90:0/2615165856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b600d2680 msgr2=0x7f6b600b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 --2- 172.21.15.90:0/2615165856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b600d2680 0x7f6b600b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 -- 172.21.15.90:0/2615165856 >> v2:172.21.15.102:3300/0 conn(0x7f6b600b2c20 msgr2=0x7f6b600cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:09.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 --2- 172.21.15.90:0/2615165856 >> v2:172.21.15.102:3300/0 conn(0x7f6b600b2c20 0x7f6b600cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 -- 172.21.15.90:0/2615165856 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b60160910 con 0x7f6b6015c120 2024-03-31T15:54:09.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5ea00640 1 --2- 172.21.15.90:0/2615165856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6015c120 0x7f6b6015fd00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b60073660 tx=0x7f6b5001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c55ebc8a8a9fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5e000640 1 -- 172.21.15.90:0/2615165856 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b500b0070 con 0x7f6b6015c120 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5e000640 1 -- 172.21.15.90:0/2615165856 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b500ab600 con 0x7f6b6015c120 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b5e000640 1 -- 172.21.15.90:0/2615165856 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b500ab9d0 con 0x7f6b6015c120 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b65600640 1 -- 172.21.15.90:0/2615165856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6015c120 msgr2=0x7f6b6015fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.154+0000 7f6b65600640 1 --2- 172.21.15.90:0/2615165856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6015c120 0x7f6b6015fd00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b60073660 tx=0x7f6b5001c3d0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 -- 172.21.15.90:0/2615165856 shutdown_connections 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 --2- 172.21.15.90:0/2615165856 >> v2:172.21.15.102:3300/0 conn(0x7f6b600b2c20 0x7f6b600cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 --2- 172.21.15.90:0/2615165856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b600d2680 0x7f6b600b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 --2- 172.21.15.90:0/2615165856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b6015c120 0x7f6b6015fd00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 -- 172.21.15.90:0/2615165856 >> 172.21.15.90:0/2615165856 conn(0x7f6b6009aa40 msgr2=0x7f6b6009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 -- 172.21.15.90:0/2615165856 shutdown_connections 2024-03-31T15:54:09.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 -- 172.21.15.90:0/2615165856 wait complete. 2024-03-31T15:54:09.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 Processor -- start 2024-03-31T15:54:09.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.158+0000 7f6b65600640 1 -- start start 2024-03-31T15:54:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 0x7f6b60156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b6015c120 0x7f6b600b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b600cea90 con 0x7f6b600d2680 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6b600ce910 con 0x7f6b600b2c20 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b600cec10 con 0x7f6b6015c120 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b5ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 0x7f6b60156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51036/0 (socket says 172.21.15.90:51036) 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b5ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 0x7f6b60156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:37178/0 (socket says 172.21.15.90:37178) 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 -- 172.21.15.90:0/1894617534 learned_addr learned my addr 172.21.15.90:0/1894617534 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b5f400640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b6015c120 0x7f6b600b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 -- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b6015c120 msgr2=0x7f6b600b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b6015c120 0x7f6b600b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 -- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 msgr2=0x7f6b60156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 0x7f6b60156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 -- 172.21.15.90:0/1894617534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b601573d0 con 0x7f6b600d2680 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b57e00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800b550 tx=0x7f6b4800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48014070 con 0x7f6b600d2680 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b48010890 con 0x7f6b600d2680 2024-03-31T15:54:09.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b601576b0 con 0x7f6b600d2680 2024-03-31T15:54:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010c80 con 0x7f6b600d2680 2024-03-31T15:54:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b60157c70 con 0x7f6b600d2680 2024-03-31T15:54:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.162+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b600686d0 con 0x7f6b600d2680 2024-03-31T15:54:09.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.166+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b48017070 con 0x7f6b600d2680 2024-03-31T15:54:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.166+0000 7f6b56a00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 0x7f6b3c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.166+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6b48086a70 con 0x7f6b600d2680 2024-03-31T15:54:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.166+0000 7f6b5ea00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 0x7f6b3c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.166+0000 7f6b5ea00640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 0x7f6b3c06a840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6b500ad000 tx=0x7f6b50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.174+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6b480566a0 con 0x7f6b600d2680 2024-03-31T15:54:09.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.334+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b600b1b60 con 0x7f6b600d2680 2024-03-31T15:54:09.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.334+0000 7f6b56a00640 1 -- 172.21.15.90:0/1894617534 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f6b48010e20 con 0x7f6b600d2680 2024-03-31T15:54:09.330 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:09.331 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 msgr2=0x7f6b3c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 0x7f6b3c06a840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6b500ad000 tx=0x7f6b50008040 comp rx=0 tx=0).stop 2024-03-31T15:54:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 msgr2=0x7f6b600b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800b550 tx=0x7f6b4800ba10 comp rx=0 tx=0).stop 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 shutdown_connections 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b3c068380 0x7f6b3c06a840 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b6015c120 0x7f6b600b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b600d2680 0x7f6b600b05d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 --2- 172.21.15.90:0/1894617534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b600b2c20 0x7f6b60156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 >> 172.21.15.90:0/1894617534 conn(0x7f6b6009aa40 msgr2=0x7f6b60159280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 shutdown_connections 2024-03-31T15:54:09.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.338+0000 7f6b65600640 1 -- 172.21.15.90:0/1894617534 wait complete. 2024-03-31T15:54:09.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:09.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 Processor -- start 2024-03-31T15:54:09.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 -- start start 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c1424a0 0x7f463c06b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f463c06bb30 0x7f463c15fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c06bfb0 con 0x7f463c0d0220 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f463c06c290 con 0x7f463c06bb30 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4642a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f463c06c570 con 0x7f463c1424a0 2024-03-31T15:54:09.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4633e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f463c06bb30 0x7f463c15fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4640a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f463c0d0220 0x7f463c15c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46354/0 (socket says 172.21.15.90:46354) 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4640a00640 1 -- 172.21.15.90:0/2338596250 learned_addr learned my addr 172.21.15.90:0/2338596250 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f463be00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c1424a0 0x7f463c06b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4633e00640 1 -- 172.21.15.90:0/2338596250 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c1424a0 msgr2=0x7f463c06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4633e00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c1424a0 0x7f463c06b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4633e00640 1 -- 172.21.15.90:0/2338596250 >> v1:172.21.15.90:6789/0 conn(0x7f463c0d0220 legacy=0x7f463c15c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f4633e00640 1 -- 172.21.15.90:0/2338596250 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c06c850 con 0x7f463c06bb30 2024-03-31T15:54:09.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.506+0000 7f463b400640 1 -- 172.21.15.90:0/2338596250 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835570019 0 0) 0x7f463c06bfb0 con 0x7f463c0d0220 2024-03-31T15:54:09.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4633e00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06bb30 0x7f463c15fbb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46240097c0 tx=0x7f462401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12a0a041949a3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f463b400640 1 -- 172.21.15.90:0/2338596250 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240b0070 con 0x7f463c06bb30 2024-03-31T15:54:09.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f463b400640 1 -- 172.21.15.90:0/2338596250 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46240ab600 con 0x7f463c06bb30 2024-03-31T15:54:09.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f463b400640 1 -- 172.21.15.90:0/2338596250 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240ab9d0 con 0x7f463c06bb30 2024-03-31T15:54:09.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- 172.21.15.90:0/2338596250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06bb30 msgr2=0x7f463c15fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06bb30 0x7f463c15fbb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46240097c0 tx=0x7f462401c4c0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- 172.21.15.90:0/2338596250 shutdown_connections 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c06bb30 0x7f463c15fbb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- 172.21.15.90:0/2338596250 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c1424a0 0x7f463c06b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- 172.21.15.90:0/2338596250 >> 172.21.15.90:0/2338596250 conn(0x7f463c0d0650 msgr2=0x7f463c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- 172.21.15.90:0/2338596250 shutdown_connections 2024-03-31T15:54:09.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- 172.21.15.90:0/2338596250 wait complete. 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 Processor -- start 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 -- start start 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c06bb30 0x7f463c06e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.514+0000 7f4642a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f463c1541a0 0x7f463c15b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f463c1692e0 con 0x7f463c0d0220 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f463c169160 con 0x7f463c1541a0 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f463c169460 con 0x7f463c06bb30 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51046/0 (socket says 172.21.15.90:51046) 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 -- 172.21.15.90:0/3026434223 learned_addr learned my addr 172.21.15.90:0/3026434223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:09.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f463be00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c06bb30 0x7f463c06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4640a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f463c1541a0 0x7f463c15b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 -- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c06bb30 msgr2=0x7f463c06e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c06bb30 0x7f463c06e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 -- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f463c1541a0 msgr2=0x7f463c15b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f463c1541a0 0x7f463c15b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 -- 172.21.15.90:0/3026434223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c15a310 con 0x7f463c0d0220 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4633e00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46240ae040 tx=0x7f4624019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240ba070 con 0x7f463c0d0220 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46240b43f0 con 0x7f463c0d0220 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46240b0040 con 0x7f463c0d0220 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c15a5f0 con 0x7f463c0d0220 2024-03-31T15:54:09.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c06efe0 con 0x7f463c0d0220 2024-03-31T15:54:09.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c0686d0 con 0x7f463c0d0220 2024-03-31T15:54:09.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.518+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46240b4800 con 0x7f463c0d0220 2024-03-31T15:54:09.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.522+0000 7f4638c00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 0x7f461806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.522+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f462411fe40 con 0x7f463c0d0220 2024-03-31T15:54:09.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.522+0000 7f463be00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 0x7f461806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.522+0000 7f463be00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 0x7f461806a870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f462c0079c0 tx=0x7f462c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.526+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f46240160b0 con 0x7f463c0d0220 2024-03-31T15:54:09.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.682+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f463c06dc40 con 0x7f463c0d0220 2024-03-31T15:54:09.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4638c00640 1 -- 172.21.15.90:0/3026434223 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f46240efa60 con 0x7f463c0d0220 2024-03-31T15:54:09.680 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:09.680 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 msgr2=0x7f461806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 0x7f461806a870 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f462c0079c0 tx=0x7f462c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 msgr2=0x7f463c1539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f46240ae040 tx=0x7f4624019260 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 shutdown_connections 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f46180683b0 0x7f461806a870 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f463c1541a0 0x7f463c15b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f463c0d0220 0x7f463c1539a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 --2- 172.21.15.90:0/3026434223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f463c06bb30 0x7f463c06e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 >> 172.21.15.90:0/3026434223 conn(0x7f463c0d0650 msgr2=0x7f463c15f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 shutdown_connections 2024-03-31T15:54:09.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.686+0000 7f4642a00640 1 -- 172.21.15.90:0/3026434223 wait complete. 2024-03-31T15:54:09.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:54:09.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 Processor -- start 2024-03-31T15:54:09.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 -- start start 2024-03-31T15:54:09.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 0x7fdad0161e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad00d0b50 con 0x7fdad00cbb70 2024-03-31T15:54:09.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdad0162510 con 0x7fdad00706c0 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad8000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdad0162db0 con 0x7fdad015e3e0 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdad00cbb70 0x7fdad00cf030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46370/0 (socket says 172.21.15.90:46370) 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad6000640 1 -- 172.21.15.90:0/2580523390 learned_addr learned my addr 172.21.15.90:0/2580523390 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad5600640 1 --2- 172.21.15.90:0/2580523390 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 0x7fdad0161e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567622411 0 0) 0x7fdad00d0b50 con 0x7fdad00cbb70 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdab8003530 con 0x7fdad00cbb70 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494669457 0 0) 0x7fdad0162510 con 0x7fdad00706c0 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdad00d0b50 con 0x7fdad00706c0 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad5600640 1 -- 172.21.15.90:0/2580523390 >> v1:172.21.15.102:6789/0 conn(0x7fdad00706c0 legacy=0x7fdad00cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad5600640 1 -- 172.21.15.90:0/2580523390 >> v1:172.21.15.90:6789/0 conn(0x7fdad00cbb70 legacy=0x7fdad00cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.850+0000 7fdad5600640 1 -- 172.21.15.90:0/2580523390 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad01627f0 con 0x7fdad015e3e0 2024-03-31T15:54:09.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad5600640 1 --2- 172.21.15.90:0/2580523390 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 0x7fdad0161e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdac40097f0 tx=0x7fdac401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978f48e8a1bc73b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40ab650 con 0x7fdad015e3e0 2024-03-31T15:54:09.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac40b0070 con 0x7fdad015e3e0 2024-03-31T15:54:09.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdacfe00640 1 -- 172.21.15.90:0/2580523390 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40aba10 con 0x7fdad015e3e0 2024-03-31T15:54:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 -- 172.21.15.90:0/2580523390 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 msgr2=0x7fdad0161e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 --2- 172.21.15.90:0/2580523390 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 0x7fdad0161e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdac40097f0 tx=0x7fdac401c450 comp rx=0 tx=0).stop 2024-03-31T15:54:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 -- 172.21.15.90:0/2580523390 shutdown_connections 2024-03-31T15:54:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 --2- 172.21.15.90:0/2580523390 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad015e3e0 0x7fdad0161e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 -- 172.21.15.90:0/2580523390 >> 172.21.15.90:0/2580523390 conn(0x7fdad00cde90 msgr2=0x7fdad00ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 -- 172.21.15.90:0/2580523390 shutdown_connections 2024-03-31T15:54:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.854+0000 7fdad8000640 1 -- 172.21.15.90:0/2580523390 wait complete. 2024-03-31T15:54:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 Processor -- start 2024-03-31T15:54:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- start start 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad00706c0 0x7fdad00c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdad015e3e0 0x7fdad00b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdad00d02e0 con 0x7fdad00cbb70 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdad00d0160 con 0x7fdad015e3e0 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdad016c1c0 con 0x7fdad00706c0 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad5600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad00706c0 0x7fdad00c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51058/0 (socket says 172.21.15.90:51058) 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 -- 172.21.15.90:0/474831919 learned_addr learned my addr 172.21.15.90:0/474831919 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad6000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdad015e3e0 0x7fdad00b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 -- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad00706c0 msgr2=0x7fdad00c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad00706c0 0x7fdad00c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 -- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdad015e3e0 msgr2=0x7fdad00b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdad015e3e0 0x7fdad00b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 -- 172.21.15.90:0/474831919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad00b6490 con 0x7fdad00cbb70 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad4c00640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00c530 tx=0x7fdabc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0052d0 con 0x7fdad00cbb70 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdabc005490 con 0x7fdad00cbb70 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc012850 con 0x7fdad00cbb70 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad00b6760 con 0x7fdad00cbb70 2024-03-31T15:54:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.858+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad01586b0 con 0x7fdad00cbb70 2024-03-31T15:54:09.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad00685f0 con 0x7fdad00cbb70 2024-03-31T15:54:09.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdabc0129f0 con 0x7fdad00cbb70 2024-03-31T15:54:09.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdacd600640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 0x7fdab006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdabc086890 con 0x7fdad00cbb70 2024-03-31T15:54:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdad5600640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 0x7fdab006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.862+0000 7fdad5600640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 0x7fdab006a840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdac40af000 tx=0x7fdac40b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:09.866+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdabc056430 con 0x7fdad00cbb70 2024-03-31T15:54:10.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.138+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdad00731a0 con 0x7fdad00cbb70 2024-03-31T15:54:10.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.834+0000 7fdacd600640 1 -- 172.21.15.90:0/474831919 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fdabc05a730 con 0x7fdad00cbb70 2024-03-31T15:54:10.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 msgr2=0x7fdab006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 0x7fdab006a840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdac40af000 tx=0x7fdac40b6040 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 msgr2=0x7fdad00b5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00c530 tx=0x7fdabc00c9f0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 shutdown_connections 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdab0068380 0x7fdab006a840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdad015e3e0 0x7fdad00b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdad00cbb70 0x7fdad00b5f50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 --2- 172.21.15.90:0/474831919 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdad00706c0 0x7fdad00c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 >> 172.21.15.90:0/474831919 conn(0x7fdad00cde90 msgr2=0x7fdad015b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 shutdown_connections 2024-03-31T15:54:10.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.838+0000 7fdad8000640 1 -- 172.21.15.90:0/474831919 wait complete. 2024-03-31T15:54:10.854 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-31T15:54:10.854 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:54:10.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 Processor -- start 2024-03-31T15:54:10.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 -- start start 2024-03-31T15:54:10.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5a3c15e4e0 0x7f5a3c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3c15e8c0 con 0x7f5a3c0ce7a0 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5a3c0d0ff0 con 0x7f5a3c15e4e0 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a41400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c0d12d0 con 0x7f5a3c06ce50 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5a3c15e4e0 0x7f5a3c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3b200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5a3c0ce7a0 0x7f5a3c0cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46374/0 (socket says 172.21.15.90:46374) 2024-03-31T15:54:10.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3b200640 1 -- 172.21.15.90:0/4120862800 learned_addr learned my addr 172.21.15.90:0/4120862800 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a39e00640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3a800640 1 -- 172.21.15.90:0/4120862800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 msgr2=0x7f5a3c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3a800640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3a800640 1 -- 172.21.15.90:0/4120862800 >> v1:172.21.15.90:6789/0 conn(0x7f5a3c0ce7a0 legacy=0x7f5a3c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a3a800640 1 -- 172.21.15.90:0/4120862800 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c0cc710 con 0x7f5a3c15e4e0 2024-03-31T15:54:10.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:10.998+0000 7f5a39400640 1 -- 172.21.15.90:0/4120862800 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2463122713 0 0) 0x7f5a3c15e8c0 con 0x7f5a3c0ce7a0 2024-03-31T15:54:10.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a3a800640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c06c910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0097c0 tx=0x7f5a2c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2bd624b14199cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:10.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a39400640 1 -- 172.21.15.90:0/4120862800 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0b0070 con 0x7f5a3c15e4e0 2024-03-31T15:54:10.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a39400640 1 -- 172.21.15.90:0/4120862800 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a2c0ab600 con 0x7f5a3c15e4e0 2024-03-31T15:54:10.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a39400640 1 -- 172.21.15.90:0/4120862800 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0ab9d0 con 0x7f5a3c15e4e0 2024-03-31T15:54:10.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 -- 172.21.15.90:0/4120862800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 msgr2=0x7f5a3c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:10.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c06c910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0097c0 tx=0x7f5a2c01c410 comp rx=0 tx=0).stop 2024-03-31T15:54:10.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 -- 172.21.15.90:0/4120862800 shutdown_connections 2024-03-31T15:54:10.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 --2- 172.21.15.90:0/4120862800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c06c910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:10.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 -- 172.21.15.90:0/4120862800 >> 172.21.15.90:0/4120862800 conn(0x7f5a3c0d1f20 msgr2=0x7f5a3c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:10.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 -- 172.21.15.90:0/4120862800 shutdown_connections 2024-03-31T15:54:10.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.002+0000 7f5a41400640 1 -- 172.21.15.90:0/4120862800 wait complete. 2024-03-31T15:54:11.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 Processor -- start 2024-03-31T15:54:11.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- start start 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c15a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a3c0ce7a0 0x7f5a3c15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0d0d90 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c0d0c10 con 0x7f5a3c0ce7a0 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c16b840 con 0x7f5a3c06ce50 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51080/0 (socket says 172.21.15.90:51080) 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 -- 172.21.15.90:0/1626544606 learned_addr learned my addr 172.21.15.90:0/1626544606 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a39e00640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a3c0ce7a0 0x7f5a3c15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3a800640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c15a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 -- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 msgr2=0x7f5a3c15a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c15a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 -- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a3c0ce7a0 msgr2=0x7f5a3c15a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a3c0ce7a0 0x7f5a3c15a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 -- 172.21.15.90:0/1626544606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c154a40 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a3b200640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c7d0 tx=0x7f5a3400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34018070 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3400a640 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400aa70 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3c1557d0 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.006+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3c155d90 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.010+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3c0685f0 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.010+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5a3400ac10 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.010+0000 7f5a32a00640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 0x7f5a1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.010+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5a340862d0 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.010+0000 7f5a3a800640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 0x7f5a1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.014+0000 7f5a3a800640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 0x7f5a1c06a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0ad000 tx=0x7f5a2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:11.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.014+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5a34055ef0 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.206+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5a3c0b2a60 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.842+0000 7f5a32a00640 1 -- 172.21.15.90:0/1626544606 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f5a3405a1f0 con 0x7f5a3c15e4e0 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 msgr2=0x7f5a1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 0x7f5a1c06a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0ad000 tx=0x7f5a2c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 msgr2=0x7f5a3c0ca420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400c7d0 tx=0x7f5a3400cc90 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 shutdown_connections 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5a1c068380 0x7f5a1c06a840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5a3c15e4e0 0x7f5a3c0ca420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a3c0ce7a0 0x7f5a3c15a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 --2- 172.21.15.90:0/1626544606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5a3c06ce50 0x7f5a3c15a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 >> 172.21.15.90:0/1626544606 conn(0x7f5a3c0d1f20 msgr2=0x7f5a3c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:11.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 shutdown_connections 2024-03-31T15:54:11.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.846+0000 7f5a41400640 1 -- 172.21.15.90:0/1626544606 wait complete. 2024-03-31T15:54:11.858 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:54:11.859 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:54:11.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:11.998+0000 7fbe58200640 1 Processor -- start 2024-03-31T15:54:11.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 -- start start 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbe5015b010 0x7fbe5015f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 0x7fbe50072820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe5006d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5015b3f0 con 0x7fbe5015b010 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5015eb10 con 0x7fbe500723c0 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe58200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbe500708b0 con 0x7fbe5006cb90 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbe5015b010 0x7fbe5015f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbe5015b010 0x7fbe5015f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51092/0 (socket says 172.21.15.90:51092) 2024-03-31T15:54:11.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 -- 172.21.15.90:0/1105556002 learned_addr learned my addr 172.21.15.90:0/1105556002 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 -- 172.21.15.90:0/1105556002 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 msgr2=0x7fbe5006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe56200640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe5006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe54e00640 1 --2- 172.21.15.90:0/1105556002 >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 0x7fbe50072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe5006d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 -- 172.21.15.90:0/1105556002 >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 msgr2=0x7fbe50072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 --2- 172.21.15.90:0/1105556002 >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 0x7fbe50072820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe55800640 1 -- 172.21.15.90:0/1105556002 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe50072f30 con 0x7fbe5015b010 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe54e00640 1 --2- 172.21.15.90:0/1105556002 >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 0x7fbe50072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:11.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.002+0000 7fbe56200640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe5006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:12.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe55800640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe5015b010 0x7fbe5015f5b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe5009f0d0 tx=0x7fbe4401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e2e16bdaf2405 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe4fe00640 1 -- 172.21.15.90:0/1105556002 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440b0070 con 0x7fbe5015b010 2024-03-31T15:54:12.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe4fe00640 1 -- 172.21.15.90:0/1105556002 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe440ab600 con 0x7fbe5015b010 2024-03-31T15:54:12.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe4fe00640 1 -- 172.21.15.90:0/1105556002 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe440ab9d0 con 0x7fbe5015b010 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 -- 172.21.15.90:0/1105556002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe5015b010 msgr2=0x7fbe5015f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe5015b010 0x7fbe5015f5b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe5009f0d0 tx=0x7fbe4401c520 comp rx=0 tx=0).stop 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 -- 172.21.15.90:0/1105556002 shutdown_connections 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe5006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 --2- 172.21.15.90:0/1105556002 >> v2:172.21.15.102:3300/0 conn(0x7fbe500723c0 0x7fbe50072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 --2- 172.21.15.90:0/1105556002 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe5015b010 0x7fbe5015f5b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 -- 172.21.15.90:0/1105556002 >> 172.21.15.90:0/1105556002 conn(0x7fbe500d13a0 msgr2=0x7fbe500d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 -- 172.21.15.90:0/1105556002 shutdown_connections 2024-03-31T15:54:12.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.006+0000 7fbe58200640 1 -- 172.21.15.90:0/1105556002 wait complete. 2024-03-31T15:54:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 Processor -- start 2024-03-31T15:54:12.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 -- start start 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe501f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe5015b010 0x7fbe50070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbe500d05d0 con 0x7fbe500723c0 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbe500d0450 con 0x7fbe5015b010 2024-03-31T15:54:12.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe58200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbe500d0750 con 0x7fbe5006cb90 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe54e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe54e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51104/0 (socket says 172.21.15.90:51104) 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.010+0000 7fbe54e00640 1 -- 172.21.15.90:0/3403014591 learned_addr learned my addr 172.21.15.90:0/3403014591 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe55800640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe501f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe56200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe5015b010 0x7fbe50070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 -- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 msgr2=0x7fbe501f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe501f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 -- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe5015b010 msgr2=0x7fbe50070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe5015b010 0x7fbe50070190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 -- 172.21.15.90:0/3403014591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe50158980 con 0x7fbe500723c0 2024-03-31T15:54:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe54e00640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00c7c0 tx=0x7fbe3c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c007a30 con 0x7fbe500723c0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3c007bd0 con 0x7fbe500723c0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0113f0 con 0x7fbe500723c0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe50158c60 con 0x7fbe500723c0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe50159220 con 0x7fbe500723c0 2024-03-31T15:54:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe500686d0 con 0x7fbe500723c0 2024-03-31T15:54:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbe3c011590 con 0x7fbe500723c0 2024-03-31T15:54:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 0x7fbe3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.014+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbe3c086f30 con 0x7fbe500723c0 2024-03-31T15:54:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.018+0000 7fbe55800640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 0x7fbe3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.018+0000 7fbe55800640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 0x7fbe3006a840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe440ad000 tx=0x7fbe44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.018+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbe3c015020 con 0x7fbe500723c0 2024-03-31T15:54:12.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.210+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe50155e60 con 0x7fbe500723c0 2024-03-31T15:54:12.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.210+0000 7fbe4d600640 1 -- 172.21.15.90:0/3403014591 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fbe3c057110 con 0x7fbe500723c0 2024-03-31T15:54:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:12.206 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:54:12.206 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 msgr2=0x7fbe3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 0x7fbe3006a840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe440ad000 tx=0x7fbe44008040 comp rx=0 tx=0).stop 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 msgr2=0x7fbe5006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00c7c0 tx=0x7fbe3c00cc80 comp rx=0 tx=0).stop 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 shutdown_connections 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe30068380 0x7fbe3006a840 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe5015b010 0x7fbe50070190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbe500723c0 0x7fbe5006fc50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 --2- 172.21.15.90:0/3403014591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbe5006cb90 0x7fbe501f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 >> 172.21.15.90:0/3403014591 conn(0x7fbe500d13a0 msgr2=0x7fbe5006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 shutdown_connections 2024-03-31T15:54:12.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.214+0000 7fbe58200640 1 -- 172.21.15.90:0/3403014591 wait complete. 2024-03-31T15:54:12.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:12.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.366+0000 7fe7ce000640 1 Processor -- start 2024-03-31T15:54:12.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.366+0000 7fe7ce000640 1 -- start start 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7ce000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7c815e1b0 0x7fe7c8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7ce000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7c806d7c0 0x7fe7c815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7ce000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c815f780 con 0x7fe7c806c7b0 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7ce000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c81622b0 con 0x7fe7c815e1b0 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7ce000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c815f1c0 con 0x7fe7c806d7c0 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7c815e1b0 0x7fe7c8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7c815e1b0 0x7fe7c8162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51118/0 (socket says 172.21.15.90:51118) 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 -- 172.21.15.90:0/2147285057 learned_addr learned my addr 172.21.15.90:0/2147285057 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7bfe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7c806c7b0 0x7fe7c806d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:33410/0 (socket says 172.21.15.90:33410) 2024-03-31T15:54:12.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7e00640 1 --2- 172.21.15.90:0/2147285057 >> v2:172.21.15.102:3300/0 conn(0x7fe7c806d7c0 0x7fe7c815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c6a00640 1 -- 172.21.15.90:0/2147285057 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339818409 0 0) 0x7fe7c815f780 con 0x7fe7c806c7b0 2024-03-31T15:54:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 -- 172.21.15.90:0/2147285057 >> v2:172.21.15.102:3300/0 conn(0x7fe7c806d7c0 msgr2=0x7fe7c815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 --2- 172.21.15.90:0/2147285057 >> v2:172.21.15.102:3300/0 conn(0x7fe7c806d7c0 0x7fe7c815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 -- 172.21.15.90:0/2147285057 >> v1:172.21.15.102:6789/0 conn(0x7fe7c806c7b0 legacy=0x7fe7c806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.370+0000 7fe7c7400640 1 -- 172.21.15.90:0/2147285057 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c815f4a0 con 0x7fe7c815e1b0 2024-03-31T15:54:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7c7400640 1 --2- 172.21.15.90:0/2147285057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c815e1b0 0x7fe7c8162740 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe7c806c5b0 tx=0x7fe7b801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeabf5517c416503 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7c6a00640 1 -- 172.21.15.90:0/2147285057 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80ab660 con 0x7fe7c815e1b0 2024-03-31T15:54:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7c6a00640 1 -- 172.21.15.90:0/2147285057 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80b0070 con 0x7fe7c815e1b0 2024-03-31T15:54:12.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7c6a00640 1 -- 172.21.15.90:0/2147285057 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80aba20 con 0x7fe7c815e1b0 2024-03-31T15:54:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 -- 172.21.15.90:0/2147285057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c815e1b0 msgr2=0x7fe7c8162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2147285057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c815e1b0 0x7fe7c8162740 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe7c806c5b0 tx=0x7fe7b801c460 comp rx=0 tx=0).stop 2024-03-31T15:54:12.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 -- 172.21.15.90:0/2147285057 shutdown_connections 2024-03-31T15:54:12.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2147285057 >> v2:172.21.15.102:3300/0 conn(0x7fe7c806d7c0 0x7fe7c815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2147285057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c815e1b0 0x7fe7c8162740 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 -- 172.21.15.90:0/2147285057 >> 172.21.15.90:0/2147285057 conn(0x7fe7c809aa40 msgr2=0x7fe7c809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 -- 172.21.15.90:0/2147285057 shutdown_connections 2024-03-31T15:54:12.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.374+0000 7fe7ce000640 1 -- 172.21.15.90:0/2147285057 wait complete. 2024-03-31T15:54:12.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 Processor -- start 2024-03-31T15:54:12.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- start start 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 0x7fe7c80c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 0x7fe7c80aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c80d2940 con 0x7fe7c806c7b0 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c80d27c0 con 0x7fe7c815e1b0 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7c80d2ac0 con 0x7fe7c806d7c0 2024-03-31T15:54:12.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51130/0 (socket says 172.21.15.90:51130) 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 -- 172.21.15.90:0/2158189842 learned_addr learned my addr 172.21.15.90:0/2158189842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7e00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 0x7fe7c80aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7bfe00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 0x7fe7c80c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 -- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 msgr2=0x7fe7c80c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 0x7fe7c80c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 -- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 msgr2=0x7fe7c80aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 0x7fe7c80aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 -- 172.21.15.90:0/2158189842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c80c7240 con 0x7fe7c806c7b0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7e00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 0x7fe7c80aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7bfe00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 0x7fe7c80c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7c7400640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8019070 tx=0x7fe7b801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80be5f0 con 0x7fe7c806c7b0 2024-03-31T15:54:12.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80be790 con 0x7fe7c806c7b0 2024-03-31T15:54:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80beb80 con 0x7fe7c806c7b0 2024-03-31T15:54:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c80c6f60 con 0x7fe7c806c7b0 2024-03-31T15:54:12.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.378+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c80b7200 con 0x7fe7c806c7b0 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c80686d0 con 0x7fe7c806c7b0 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe7b80b43f0 con 0x7fe7c806c7b0 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7bf400640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 0x7fe7a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe7b8121770 con 0x7fe7c806c7b0 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7bfe00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 0x7fe7a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.382+0000 7fe7bfe00640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 0x7fe7a406a840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.386+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe7b8016110 con 0x7fe7c806c7b0 2024-03-31T15:54:12.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.610+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7c80c81c0 con 0x7fe7c806c7b0 2024-03-31T15:54:12.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7bf400640 1 -- 172.21.15.90:0/2158189842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7fe7b80f1420 con 0x7fe7c806c7b0 2024-03-31T15:54:12.608 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:12.609 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:12.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 msgr2=0x7fe7a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 0x7fe7a406a840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0007fd0 tx=0x7fe7b0009040 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 msgr2=0x7fe7c80b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8019070 tx=0x7fe7b801aca0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 shutdown_connections 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7a4068380 0x7fe7a406a840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7c815e1b0 0x7fe7c80aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7c806d7c0 0x7fe7c80c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 --2- 172.21.15.90:0/2158189842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7c806c7b0 0x7fe7c80b7a00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 >> 172.21.15.90:0/2158189842 conn(0x7fe7c809aa40 msgr2=0x7fe7c806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 shutdown_connections 2024-03-31T15:54:12.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.614+0000 7fe7ce000640 1 -- 172.21.15.90:0/2158189842 wait complete. 2024-03-31T15:54:12.632 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:54:12.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:54:12.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.770+0000 7f14b3200640 1 Processor -- start 2024-03-31T15:54:12.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.770+0000 7f14b3200640 1 -- start start 2024-03-31T15:54:12.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b3200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b3200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 0x7f14b406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b3200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b406de50 con 0x7f14b41457e0 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b3200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f14b406d380 con 0x7f14b4160470 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b3200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f14b41608e0 con 0x7f14b40d0680 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 0x7f14b406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:36238/0 (socket says 172.21.15.90:36238) 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 -- 172.21.15.90:0/3124258492 learned_addr learned my addr 172.21.15.90:0/3124258492 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b1e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f14b41457e0 0x7f14b40cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:33426/0 (socket says 172.21.15.90:33426) 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14b2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 0x7f14b406ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:37250/0 (socket says 172.21.15.90:37250) 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 -- 172.21.15.90:0/3124258492 >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 msgr2=0x7f14b406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 --2- 172.21.15.90:0/3124258492 >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 0x7f14b406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 -- 172.21.15.90:0/3124258492 >> v1:172.21.15.102:6789/0 conn(0x7f14b41457e0 legacy=0x7f14b40cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.774+0000 7f14a9400640 1 -- 172.21.15.90:0/3124258492 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b406d5b0 con 0x7f14b40d0680 2024-03-31T15:54:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14a9400640 1 --2- 172.21.15.90:0/3124258492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f149c009870 tx=0x7f149c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d19dea8366aa7c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b1400640 1 -- 172.21.15.90:0/3124258492 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c0ab650 con 0x7f14b40d0680 2024-03-31T15:54:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b1400640 1 -- 172.21.15.90:0/3124258492 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f149c0b0070 con 0x7f14b40d0680 2024-03-31T15:54:12.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b1400640 1 -- 172.21.15.90:0/3124258492 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c0aba10 con 0x7f14b40d0680 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- 172.21.15.90:0/3124258492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 msgr2=0x7f14b40b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 --2- 172.21.15.90:0/3124258492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f149c009870 tx=0x7f149c01c560 comp rx=0 tx=0).stop 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- 172.21.15.90:0/3124258492 shutdown_connections 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 --2- 172.21.15.90:0/3124258492 >> v2:172.21.15.102:3300/0 conn(0x7f14b4160470 0x7f14b406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 --2- 172.21.15.90:0/3124258492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b40d0680 0x7f14b40b2a80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- 172.21.15.90:0/3124258492 >> 172.21.15.90:0/3124258492 conn(0x7f14b406b9f0 msgr2=0x7f14b406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- 172.21.15.90:0/3124258492 shutdown_connections 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- 172.21.15.90:0/3124258492 wait complete. 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 Processor -- start 2024-03-31T15:54:12.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.778+0000 7f14b3200640 1 -- start start 2024-03-31T15:54:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b41457e0 0x7f14b4072280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14b4160470 0x7f14b412b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f14b40cf0f0 con 0x7f14b40d0680 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f14b40cef70 con 0x7f14b4160470 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f14b40cf270 con 0x7f14b41457e0 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51136/0 (socket says 172.21.15.90:51136) 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 -- 172.21.15.90:0/2580014085 learned_addr learned my addr 172.21.15.90:0/2580014085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14a9400640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b41457e0 0x7f14b4072280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b2800640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14b4160470 0x7f14b412b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 -- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b41457e0 msgr2=0x7f14b4072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b41457e0 0x7f14b4072280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 -- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14b4160470 msgr2=0x7f14b412b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14b4160470 0x7f14b412b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 -- 172.21.15.90:0/2580014085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b40727c0 con 0x7f14b40d0680 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b1e00640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c7c0 tx=0x7f14a400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4017070 con 0x7f14b40d0680 2024-03-31T15:54:12.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a400a640 con 0x7f14b40d0680 2024-03-31T15:54:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a400a9f0 con 0x7f14b40d0680 2024-03-31T15:54:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b413fa00 con 0x7f14b40d0680 2024-03-31T15:54:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b413ffc0 con 0x7f14b40d0680 2024-03-31T15:54:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.782+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b40685f0 con 0x7f14b40d0680 2024-03-31T15:54:12.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.786+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f14a401a070 con 0x7f14b40d0680 2024-03-31T15:54:12.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.786+0000 7f14aaa00640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 0x7f149006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:12.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.786+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f14a4086220 con 0x7f14b40d0680 2024-03-31T15:54:12.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.786+0000 7f14a9400640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 0x7f149006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:12.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.786+0000 7f14a9400640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 0x7f149006a840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f149c0af000 tx=0x7f149c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:12.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.790+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f14b40727c0 con 0x7f14b40d0680 2024-03-31T15:54:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.946+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14b41667b0 con 0x7f1490068380 2024-03-31T15:54:12.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.958+0000 7f14aaa00640 1 -- 172.21.15.90:0/2580014085 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332562 (secure 0 0 0) 0x7f14b41667b0 con 0x7f1490068380 2024-03-31T15:54:12.955 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:12.957 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:54:12.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 msgr2=0x7f149006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 0x7f149006a840 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f149c0af000 tx=0x7f149c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 msgr2=0x7f14b413c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f14a400c7c0 tx=0x7f14a400cc80 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 shutdown_connections 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1490068380 0x7f149006a840 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14b4160470 0x7f14b412b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f14b41457e0 0x7f14b4072280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 --2- 172.21.15.90:0/2580014085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f14b40d0680 0x7f14b413c2e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 >> 172.21.15.90:0/2580014085 conn(0x7f14b406b9f0 msgr2=0x7f14b40cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 shutdown_connections 2024-03-31T15:54:12.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:12.962+0000 7f14b3200640 1 -- 172.21.15.90:0/2580014085 wait complete. 2024-03-31T15:54:12.976 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-31T15:54:11.220929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219416,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755304,"statfs":{"total":773094113280,"available":772869431296,"internally_reserved":0,"allocated":3317760,"data_stored":2007160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003885"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359793+0000","last_change":"2024-03-31T15:53:46.757375+0000","last_active":"2024-03-31T15:53:53.359793+0000","last_peered":"2024-03-31T15:53:53.359793+0000","last_clean":"2024-03-31T15:53:53.359793+0000","last_became_active":"2024-03-31T15:53:46.757092+0000","last_became_peered":"2024-03-31T15:53:46.757092+0000","last_unstale":"2024-03-31T15:53:53.359793+0000","last_undegraded":"2024-03-31T15:53:53.359793+0000","last_fullsized":"2024-03-31T15:53:53.359793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248347+0000","last_change":"2024-03-31T15:53:47.766865+0000","last_active":"2024-03-31T15:53:53.248347+0000","last_peered":"2024-03-31T15:53:53.248347+0000","last_clean":"2024-03-31T15:53:53.248347+0000","last_became_active":"2024-03-31T15:53:47.766202+0000","last_became_peered":"2024-03-31T15:53:47.766202+0000","last_unstale":"2024-03-31T15:53:53.248347+0000","last_undegraded":"2024-03-31T15:53:53.248347+0000","last_fullsized":"2024-03-31T15:53:53.248347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:31.997737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247733+0000","last_change":"2024-03-31T15:53:51.804152+0000","last_active":"2024-03-31T15:53:53.247733+0000","last_peered":"2024-03-31T15:53:53.247733+0000","last_clean":"2024-03-31T15:53:53.247733+0000","last_became_active":"2024-03-31T15:53:51.802511+0000","last_became_peered":"2024-03-31T15:53:51.802511+0000","last_unstale":"2024-03-31T15:53:53.247733+0000","last_undegraded":"2024-03-31T15:53:53.247733+0000","last_fullsized":"2024-03-31T15:53:53.247733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:38.965494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248547+0000","last_change":"2024-03-31T15:53:46.763714+0000","last_active":"2024-03-31T15:53:53.248547+0000","last_peered":"2024-03-31T15:53:53.248547+0000","last_clean":"2024-03-31T15:53:53.248547+0000","last_became_active":"2024-03-31T15:53:46.763376+0000","last_became_peered":"2024-03-31T15:53:46.763376+0000","last_unstale":"2024-03-31T15:53:53.248547+0000","last_undegraded":"2024-03-31T15:53:53.248547+0000","last_fullsized":"2024-03-31T15:53:53.248547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359675+0000","last_change":"2024-03-31T15:53:47.771438+0000","last_active":"2024-03-31T15:53:53.359675+0000","last_peered":"2024-03-31T15:53:53.359675+0000","last_clean":"2024-03-31T15:53:53.359675+0000","last_became_active":"2024-03-31T15:53:47.771175+0000","last_became_peered":"2024-03-31T15:53:47.771175+0000","last_unstale":"2024-03-31T15:53:53.359675+0000","last_undegraded":"2024-03-31T15:53:53.359675+0000","last_fullsized":"2024-03-31T15:53:53.359675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792499+0000","last_change":"2024-03-31T15:53:51.797656+0000","last_active":"2024-03-31T15:53:52.792499+0000","last_peered":"2024-03-31T15:53:52.792499+0000","last_clean":"2024-03-31T15:53:52.792499+0000","last_became_active":"2024-03-31T15:53:51.797356+0000","last_became_peered":"2024-03-31T15:53:51.797356+0000","last_unstale":"2024-03-31T15:53:52.792499+0000","last_undegraded":"2024-03-31T15:53:52.792499+0000","last_fullsized":"2024-03-31T15:53:52.792499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:05.027492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792090+0000","last_change":"2024-03-31T15:53:46.764787+0000","last_active":"2024-03-31T15:53:52.792090+0000","last_peered":"2024-03-31T15:53:52.792090+0000","last_clean":"2024-03-31T15:53:52.792090+0000","last_became_active":"2024-03-31T15:53:46.764484+0000","last_became_peered":"2024-03-31T15:53:46.764484+0000","last_unstale":"2024-03-31T15:53:52.792090+0000","last_undegraded":"2024-03-31T15:53:52.792090+0000","last_fullsized":"2024-03-31T15:53:52.792090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:15.262251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789729+0000","last_change":"2024-03-31T15:53:47.773460+0000","last_active":"2024-03-31T15:53:52.789729+0000","last_peered":"2024-03-31T15:53:52.789729+0000","last_clean":"2024-03-31T15:53:52.789729+0000","last_became_active":"2024-03-31T15:53:47.773148+0000","last_became_peered":"2024-03-31T15:53:47.773148+0000","last_unstale":"2024-03-31T15:53:52.789729+0000","last_undegraded":"2024-03-31T15:53:52.789729+0000","last_fullsized":"2024-03-31T15:53:52.789729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.396848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793233+0000","last_change":"2024-03-31T15:53:51.799743+0000","last_active":"2024-03-31T15:53:52.793233+0000","last_peered":"2024-03-31T15:53:52.793233+0000","last_clean":"2024-03-31T15:53:52.793233+0000","last_became_active":"2024-03-31T15:53:51.799332+0000","last_became_peered":"2024-03-31T15:53:51.799332+0000","last_unstale":"2024-03-31T15:53:52.793233+0000","last_undegraded":"2024-03-31T15:53:52.793233+0000","last_fullsized":"2024-03-31T15:53:52.793233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:58.431101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791817+0000","last_change":"2024-03-31T15:53:46.765429+0000","last_active":"2024-03-31T15:53:52.791817+0000","last_peered":"2024-03-31T15:53:52.791817+0000","last_clean":"2024-03-31T15:53:52.791817+0000","last_became_active":"2024-03-31T15:53:46.765129+0000","last_became_peered":"2024-03-31T15:53:46.765129+0000","last_unstale":"2024-03-31T15:53:52.791817+0000","last_undegraded":"2024-03-31T15:53:52.791817+0000","last_fullsized":"2024-03-31T15:53:52.791817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:00:22.460897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790207+0000","last_change":"2024-03-31T15:53:47.770281+0000","last_active":"2024-03-31T15:53:52.790207+0000","last_peered":"2024-03-31T15:53:52.790207+0000","last_clean":"2024-03-31T15:53:52.790207+0000","last_became_active":"2024-03-31T15:53:47.769991+0000","last_became_peered":"2024-03-31T15:53:47.769991+0000","last_unstale":"2024-03-31T15:53:52.790207+0000","last_undegraded":"2024-03-31T15:53:52.790207+0000","last_fullsized":"2024-03-31T15:53:52.790207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:20.227687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792734+0000","last_change":"2024-03-31T15:53:51.797162+0000","last_active":"2024-03-31T15:53:52.792734+0000","last_peered":"2024-03-31T15:53:52.792734+0000","last_clean":"2024-03-31T15:53:52.792734+0000","last_became_active":"2024-03-31T15:53:51.796702+0000","last_became_peered":"2024-03-31T15:53:51.796702+0000","last_unstale":"2024-03-31T15:53:52.792734+0000","last_undegraded":"2024-03-31T15:53:52.792734+0000","last_fullsized":"2024-03-31T15:53:52.792734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:40.593467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306348+0000","last_change":"2024-03-31T15:53:47.762689+0000","last_active":"2024-03-31T15:53:53.306348+0000","last_peered":"2024-03-31T15:53:53.306348+0000","last_clean":"2024-03-31T15:53:53.306348+0000","last_became_active":"2024-03-31T15:53:47.762427+0000","last_became_peered":"2024-03-31T15:53:47.762427+0000","last_unstale":"2024-03-31T15:53:53.306348+0000","last_undegraded":"2024-03-31T15:53:53.306348+0000","last_fullsized":"2024-03-31T15:53:53.306348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791816+0000","last_change":"2024-03-31T15:53:46.757859+0000","last_active":"2024-03-31T15:53:52.791816+0000","last_peered":"2024-03-31T15:53:52.791816+0000","last_clean":"2024-03-31T15:53:52.791816+0000","last_became_active":"2024-03-31T15:53:46.757618+0000","last_became_peered":"2024-03-31T15:53:46.757618+0000","last_unstale":"2024-03-31T15:53:52.791816+0000","last_undegraded":"2024-03-31T15:53:52.791816+0000","last_fullsized":"2024-03-31T15:53:52.791816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:08:18.634688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793519+0000","last_change":"2024-03-31T15:53:51.797915+0000","last_active":"2024-03-31T15:53:52.793519+0000","last_peered":"2024-03-31T15:53:52.793519+0000","last_clean":"2024-03-31T15:53:52.793519+0000","last_became_active":"2024-03-31T15:53:51.797515+0000","last_became_peered":"2024-03-31T15:53:51.797515+0000","last_unstale":"2024-03-31T15:53:52.793519+0000","last_undegraded":"2024-03-31T15:53:52.793519+0000","last_fullsized":"2024-03-31T15:53:52.793519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247998+0000","last_change":"2024-03-31T15:53:47.765185+0000","last_active":"2024-03-31T15:53:53.247998+0000","last_peered":"2024-03-31T15:53:53.247998+0000","last_clean":"2024-03-31T15:53:53.247998+0000","last_became_active":"2024-03-31T15:53:47.764777+0000","last_became_peered":"2024-03-31T15:53:47.764777+0000","last_unstale":"2024-03-31T15:53:53.247998+0000","last_undegraded":"2024-03-31T15:53:53.247998+0000","last_fullsized":"2024-03-31T15:53:53.247998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:23.224004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920654+0000","last_change":"2024-03-31T15:53:46.762534+0000","last_active":"2024-03-31T15:53:52.920654+0000","last_peered":"2024-03-31T15:53:52.920654+0000","last_clean":"2024-03-31T15:53:52.920654+0000","last_became_active":"2024-03-31T15:53:46.762081+0000","last_became_peered":"2024-03-31T15:53:46.762081+0000","last_unstale":"2024-03-31T15:53:52.920654+0000","last_undegraded":"2024-03-31T15:53:52.920654+0000","last_fullsized":"2024-03-31T15:53:52.920654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793296+0000","last_change":"2024-03-31T15:53:51.802744+0000","last_active":"2024-03-31T15:53:52.793296+0000","last_peered":"2024-03-31T15:53:52.793296+0000","last_clean":"2024-03-31T15:53:52.793296+0000","last_became_active":"2024-03-31T15:53:51.802328+0000","last_became_peered":"2024-03-31T15:53:51.802328+0000","last_unstale":"2024-03-31T15:53:52.793296+0000","last_undegraded":"2024-03-31T15:53:52.793296+0000","last_fullsized":"2024-03-31T15:53:52.793296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:00.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247904+0000","last_change":"2024-03-31T15:53:47.751398+0000","last_active":"2024-03-31T15:53:53.247904+0000","last_peered":"2024-03-31T15:53:53.247904+0000","last_clean":"2024-03-31T15:53:53.247904+0000","last_became_active":"2024-03-31T15:53:47.750758+0000","last_became_peered":"2024-03-31T15:53:47.750758+0000","last_unstale":"2024-03-31T15:53:53.247904+0000","last_undegraded":"2024-03-31T15:53:53.247904+0000","last_fullsized":"2024-03-31T15:53:53.247904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:31.275051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920257+0000","last_change":"2024-03-31T15:53:46.755068+0000","last_active":"2024-03-31T15:53:52.920257+0000","last_peered":"2024-03-31T15:53:52.920257+0000","last_clean":"2024-03-31T15:53:52.920257+0000","last_became_active":"2024-03-31T15:53:46.754754+0000","last_became_peered":"2024-03-31T15:53:46.754754+0000","last_unstale":"2024-03-31T15:53:52.920257+0000","last_undegraded":"2024-03-31T15:53:52.920257+0000","last_fullsized":"2024-03-31T15:53:52.920257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921340+0000","last_change":"2024-03-31T15:53:51.803182+0000","last_active":"2024-03-31T15:53:52.921340+0000","last_peered":"2024-03-31T15:53:52.921340+0000","last_clean":"2024-03-31T15:53:52.921340+0000","last_became_active":"2024-03-31T15:53:51.802710+0000","last_became_peered":"2024-03-31T15:53:51.802710+0000","last_unstale":"2024-03-31T15:53:52.921340+0000","last_undegraded":"2024-03-31T15:53:52.921340+0000","last_fullsized":"2024-03-31T15:53:52.921340+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:40.165393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792965+0000","last_change":"2024-03-31T15:53:47.755358+0000","last_active":"2024-03-31T15:53:52.792965+0000","last_peered":"2024-03-31T15:53:52.792965+0000","last_clean":"2024-03-31T15:53:52.792965+0000","last_became_active":"2024-03-31T15:53:47.754487+0000","last_became_peered":"2024-03-31T15:53:47.754487+0000","last_unstale":"2024-03-31T15:53:52.792965+0000","last_undegraded":"2024-03-31T15:53:52.792965+0000","last_fullsized":"2024-03-31T15:53:52.792965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:53.949585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791156+0000","last_change":"2024-03-31T15:53:46.755464+0000","last_active":"2024-03-31T15:53:52.791156+0000","last_peered":"2024-03-31T15:53:52.791156+0000","last_clean":"2024-03-31T15:53:52.791156+0000","last_became_active":"2024-03-31T15:53:46.754626+0000","last_became_peered":"2024-03-31T15:53:46.754626+0000","last_unstale":"2024-03-31T15:53:52.791156+0000","last_undegraded":"2024-03-31T15:53:52.791156+0000","last_fullsized":"2024-03-31T15:53:52.791156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:18.769264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793394+0000","last_change":"2024-03-31T15:53:51.803474+0000","last_active":"2024-03-31T15:53:52.793394+0000","last_peered":"2024-03-31T15:53:52.793394+0000","last_clean":"2024-03-31T15:53:52.793394+0000","last_became_active":"2024-03-31T15:53:51.803014+0000","last_became_peered":"2024-03-31T15:53:51.803014+0000","last_unstale":"2024-03-31T15:53:52.793394+0000","last_undegraded":"2024-03-31T15:53:52.793394+0000","last_fullsized":"2024-03-31T15:53:52.793394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359741+0000","last_change":"2024-03-31T15:53:47.754929+0000","last_active":"2024-03-31T15:53:53.359741+0000","last_peered":"2024-03-31T15:53:53.359741+0000","last_clean":"2024-03-31T15:53:53.359741+0000","last_became_active":"2024-03-31T15:53:47.754625+0000","last_became_peered":"2024-03-31T15:53:47.754625+0000","last_unstale":"2024-03-31T15:53:53.359741+0000","last_undegraded":"2024-03-31T15:53:53.359741+0000","last_fullsized":"2024-03-31T15:53:53.359741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307339+0000","last_change":"2024-03-31T15:53:46.757784+0000","last_active":"2024-03-31T15:53:53.307339+0000","last_peered":"2024-03-31T15:53:53.307339+0000","last_clean":"2024-03-31T15:53:53.307339+0000","last_became_active":"2024-03-31T15:53:46.757342+0000","last_became_peered":"2024-03-31T15:53:46.757342+0000","last_unstale":"2024-03-31T15:53:53.307339+0000","last_undegraded":"2024-03-31T15:53:53.307339+0000","last_fullsized":"2024-03-31T15:53:53.307339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792725+0000","last_change":"2024-03-31T15:53:51.800310+0000","last_active":"2024-03-31T15:53:52.792725+0000","last_peered":"2024-03-31T15:53:52.792725+0000","last_clean":"2024-03-31T15:53:52.792725+0000","last_became_active":"2024-03-31T15:53:51.798223+0000","last_became_peered":"2024-03-31T15:53:51.798223+0000","last_unstale":"2024-03-31T15:53:52.792725+0000","last_undegraded":"2024-03-31T15:53:52.792725+0000","last_fullsized":"2024-03-31T15:53:52.792725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248692+0000","last_change":"2024-03-31T15:53:47.754983+0000","last_active":"2024-03-31T15:53:53.248692+0000","last_peered":"2024-03-31T15:53:53.248692+0000","last_clean":"2024-03-31T15:53:53.248692+0000","last_became_active":"2024-03-31T15:53:47.754734+0000","last_became_peered":"2024-03-31T15:53:47.754734+0000","last_unstale":"2024-03-31T15:53:53.248692+0000","last_undegraded":"2024-03-31T15:53:53.248692+0000","last_fullsized":"2024-03-31T15:53:53.248692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:31:24.931497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921179+0000","last_change":"2024-03-31T15:53:46.763360+0000","last_active":"2024-03-31T15:53:52.921179+0000","last_peered":"2024-03-31T15:53:52.921179+0000","last_clean":"2024-03-31T15:53:52.921179+0000","last_became_active":"2024-03-31T15:53:46.763005+0000","last_became_peered":"2024-03-31T15:53:46.763005+0000","last_unstale":"2024-03-31T15:53:52.921179+0000","last_undegraded":"2024-03-31T15:53:52.921179+0000","last_fullsized":"2024-03-31T15:53:52.921179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:30.435282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247663+0000","last_change":"2024-03-31T15:53:51.800117+0000","last_active":"2024-03-31T15:53:53.247663+0000","last_peered":"2024-03-31T15:53:53.247663+0000","last_clean":"2024-03-31T15:53:53.247663+0000","last_became_active":"2024-03-31T15:53:51.799217+0000","last_became_peered":"2024-03-31T15:53:51.799217+0000","last_unstale":"2024-03-31T15:53:53.247663+0000","last_undegraded":"2024-03-31T15:53:53.247663+0000","last_fullsized":"2024-03-31T15:53:53.247663+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:39.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920855+0000","last_change":"2024-03-31T15:53:47.754428+0000","last_active":"2024-03-31T15:53:52.920855+0000","last_peered":"2024-03-31T15:53:52.920855+0000","last_clean":"2024-03-31T15:53:52.920855+0000","last_became_active":"2024-03-31T15:53:47.753085+0000","last_became_peered":"2024-03-31T15:53:47.753085+0000","last_unstale":"2024-03-31T15:53:52.920855+0000","last_undegraded":"2024-03-31T15:53:52.920855+0000","last_fullsized":"2024-03-31T15:53:52.920855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792251+0000","last_change":"2024-03-31T15:53:46.763802+0000","last_active":"2024-03-31T15:53:52.792251+0000","last_peered":"2024-03-31T15:53:52.792251+0000","last_clean":"2024-03-31T15:53:52.792251+0000","last_became_active":"2024-03-31T15:53:46.762025+0000","last_became_peered":"2024-03-31T15:53:46.762025+0000","last_unstale":"2024-03-31T15:53:52.792251+0000","last_undegraded":"2024-03-31T15:53:52.792251+0000","last_fullsized":"2024-03-31T15:53:52.792251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:50.235479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359173+0000","last_change":"2024-03-31T15:53:51.796387+0000","last_active":"2024-03-31T15:53:53.359173+0000","last_peered":"2024-03-31T15:53:53.359173+0000","last_clean":"2024-03-31T15:53:53.359173+0000","last_became_active":"2024-03-31T15:53:51.795890+0000","last_became_peered":"2024-03-31T15:53:51.795890+0000","last_unstale":"2024-03-31T15:53:53.359173+0000","last_undegraded":"2024-03-31T15:53:53.359173+0000","last_fullsized":"2024-03-31T15:53:53.359173+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790297+0000","last_change":"2024-03-31T15:53:47.765672+0000","last_active":"2024-03-31T15:53:52.790297+0000","last_peered":"2024-03-31T15:53:52.790297+0000","last_clean":"2024-03-31T15:53:52.790297+0000","last_became_active":"2024-03-31T15:53:47.765481+0000","last_became_peered":"2024-03-31T15:53:47.765481+0000","last_unstale":"2024-03-31T15:53:52.790297+0000","last_undegraded":"2024-03-31T15:53:52.790297+0000","last_fullsized":"2024-03-31T15:53:52.790297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:14.332486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920943+0000","last_change":"2024-03-31T15:53:46.752123+0000","last_active":"2024-03-31T15:53:52.920943+0000","last_peered":"2024-03-31T15:53:52.920943+0000","last_clean":"2024-03-31T15:53:52.920943+0000","last_became_active":"2024-03-31T15:53:46.751752+0000","last_became_peered":"2024-03-31T15:53:46.751752+0000","last_unstale":"2024-03-31T15:53:52.920943+0000","last_undegraded":"2024-03-31T15:53:52.920943+0000","last_fullsized":"2024-03-31T15:53:52.920943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920358+0000","last_change":"2024-03-31T15:53:51.798289+0000","last_active":"2024-03-31T15:53:52.920358+0000","last_peered":"2024-03-31T15:53:52.920358+0000","last_clean":"2024-03-31T15:53:52.920358+0000","last_became_active":"2024-03-31T15:53:51.797843+0000","last_became_peered":"2024-03-31T15:53:51.797843+0000","last_unstale":"2024-03-31T15:53:52.920358+0000","last_undegraded":"2024-03-31T15:53:52.920358+0000","last_fullsized":"2024-03-31T15:53:52.920358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793832+0000","last_change":"2024-03-31T15:53:47.746865+0000","last_active":"2024-03-31T15:53:52.793832+0000","last_peered":"2024-03-31T15:53:52.793832+0000","last_clean":"2024-03-31T15:53:52.793832+0000","last_became_active":"2024-03-31T15:53:47.746098+0000","last_became_peered":"2024-03-31T15:53:47.746098+0000","last_unstale":"2024-03-31T15:53:52.793832+0000","last_undegraded":"2024-03-31T15:53:52.793832+0000","last_fullsized":"2024-03-31T15:53:52.793832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:31.443157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790702+0000","last_change":"2024-03-31T15:53:46.753263+0000","last_active":"2024-03-31T15:53:52.790702+0000","last_peered":"2024-03-31T15:53:52.790702+0000","last_clean":"2024-03-31T15:53:52.790702+0000","last_became_active":"2024-03-31T15:53:46.751947+0000","last_became_peered":"2024-03-31T15:53:46.751947+0000","last_unstale":"2024-03-31T15:53:52.790702+0000","last_undegraded":"2024-03-31T15:53:52.790702+0000","last_fullsized":"2024-03-31T15:53:52.790702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920882+0000","last_change":"2024-03-31T15:53:51.797078+0000","last_active":"2024-03-31T15:53:52.920882+0000","last_peered":"2024-03-31T15:53:52.920882+0000","last_clean":"2024-03-31T15:53:52.920882+0000","last_became_active":"2024-03-31T15:53:51.796451+0000","last_became_peered":"2024-03-31T15:53:51.796451+0000","last_unstale":"2024-03-31T15:53:52.920882+0000","last_undegraded":"2024-03-31T15:53:52.920882+0000","last_fullsized":"2024-03-31T15:53:52.920882+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920377+0000","last_change":"2024-03-31T15:53:47.752652+0000","last_active":"2024-03-31T15:53:52.920377+0000","last_peered":"2024-03-31T15:53:52.920377+0000","last_clean":"2024-03-31T15:53:52.920377+0000","last_became_active":"2024-03-31T15:53:47.752105+0000","last_became_peered":"2024-03-31T15:53:47.752105+0000","last_unstale":"2024-03-31T15:53:52.920377+0000","last_undegraded":"2024-03-31T15:53:52.920377+0000","last_fullsized":"2024-03-31T15:53:52.920377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528393+0000","last_change":"2024-03-31T15:53:46.753745+0000","last_active":"2024-03-31T15:53:53.528393+0000","last_peered":"2024-03-31T15:53:53.528393+0000","last_clean":"2024-03-31T15:53:53.528393+0000","last_became_active":"2024-03-31T15:53:46.753423+0000","last_became_peered":"2024-03-31T15:53:46.753423+0000","last_unstale":"2024-03-31T15:53:53.528393+0000","last_undegraded":"2024-03-31T15:53:53.528393+0000","last_fullsized":"2024-03-31T15:53:53.528393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921329+0000","last_change":"2024-03-31T15:53:51.802368+0000","last_active":"2024-03-31T15:53:52.921329+0000","last_peered":"2024-03-31T15:53:52.921329+0000","last_clean":"2024-03-31T15:53:52.921329+0000","last_became_active":"2024-03-31T15:53:51.801946+0000","last_became_peered":"2024-03-31T15:53:51.801946+0000","last_unstale":"2024-03-31T15:53:52.921329+0000","last_undegraded":"2024-03-31T15:53:52.921329+0000","last_fullsized":"2024-03-31T15:53:52.921329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.358775+0000","last_change":"2024-03-31T15:53:47.745539+0000","last_active":"2024-03-31T15:53:53.358775+0000","last_peered":"2024-03-31T15:53:53.358775+0000","last_clean":"2024-03-31T15:53:53.358775+0000","last_became_active":"2024-03-31T15:53:47.745265+0000","last_became_peered":"2024-03-31T15:53:47.745265+0000","last_unstale":"2024-03-31T15:53:53.358775+0000","last_undegraded":"2024-03-31T15:53:53.358775+0000","last_fullsized":"2024-03-31T15:53:53.358775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:12:22.550188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791245+0000","last_change":"2024-03-31T15:53:46.761890+0000","last_active":"2024-03-31T15:53:52.791245+0000","last_peered":"2024-03-31T15:53:52.791245+0000","last_clean":"2024-03-31T15:53:52.791245+0000","last_became_active":"2024-03-31T15:53:46.761417+0000","last_became_peered":"2024-03-31T15:53:46.761417+0000","last_unstale":"2024-03-31T15:53:52.791245+0000","last_undegraded":"2024-03-31T15:53:52.791245+0000","last_fullsized":"2024-03-31T15:53:52.791245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:25.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359685+0000","last_change":"2024-03-31T15:53:51.796610+0000","last_active":"2024-03-31T15:53:53.359685+0000","last_peered":"2024-03-31T15:53:53.359685+0000","last_clean":"2024-03-31T15:53:53.359685+0000","last_became_active":"2024-03-31T15:53:51.796112+0000","last_became_peered":"2024-03-31T15:53:51.796112+0000","last_unstale":"2024-03-31T15:53:53.359685+0000","last_undegraded":"2024-03-31T15:53:53.359685+0000","last_fullsized":"2024-03-31T15:53:53.359685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247912+0000","last_change":"2024-03-31T15:53:47.768681+0000","last_active":"2024-03-31T15:53:53.247912+0000","last_peered":"2024-03-31T15:53:53.247912+0000","last_clean":"2024-03-31T15:53:53.247912+0000","last_became_active":"2024-03-31T15:53:47.768416+0000","last_became_peered":"2024-03-31T15:53:47.768416+0000","last_unstale":"2024-03-31T15:53:53.247912+0000","last_undegraded":"2024-03-31T15:53:53.247912+0000","last_fullsized":"2024-03-31T15:53:53.247912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:02.383327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789855+0000","last_change":"2024-03-31T15:53:46.760553+0000","last_active":"2024-03-31T15:53:52.789855+0000","last_peered":"2024-03-31T15:53:52.789855+0000","last_clean":"2024-03-31T15:53:52.789855+0000","last_became_active":"2024-03-31T15:53:46.760258+0000","last_became_peered":"2024-03-31T15:53:46.760258+0000","last_unstale":"2024-03-31T15:53:52.789855+0000","last_undegraded":"2024-03-31T15:53:52.789855+0000","last_fullsized":"2024-03-31T15:53:52.789855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791305+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:53:52.791305+0000","last_peered":"2024-03-31T15:53:52.791305+0000","last_clean":"2024-03-31T15:53:52.791305+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:53:52.791305+0000","last_undegraded":"2024-03-31T15:53:52.791305+0000","last_fullsized":"2024-03-31T15:53:52.791305+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306820+0000","last_change":"2024-03-31T15:53:51.789901+0000","last_active":"2024-03-31T15:53:53.306820+0000","last_peered":"2024-03-31T15:53:53.306820+0000","last_clean":"2024-03-31T15:53:53.306820+0000","last_became_active":"2024-03-31T15:53:51.789529+0000","last_became_peered":"2024-03-31T15:53:51.789529+0000","last_unstale":"2024-03-31T15:53:53.306820+0000","last_undegraded":"2024-03-31T15:53:53.306820+0000","last_fullsized":"2024-03-31T15:53:53.306820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791956+0000","last_change":"2024-03-31T15:53:47.764710+0000","last_active":"2024-03-31T15:53:52.791956+0000","last_peered":"2024-03-31T15:53:52.791956+0000","last_clean":"2024-03-31T15:53:52.791956+0000","last_became_active":"2024-03-31T15:53:47.764408+0000","last_became_peered":"2024-03-31T15:53:47.764408+0000","last_unstale":"2024-03-31T15:53:52.791956+0000","last_undegraded":"2024-03-31T15:53:52.791956+0000","last_fullsized":"2024-03-31T15:53:52.791956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:14:45.512051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528719+0000","last_change":"2024-03-31T15:53:46.757343+0000","last_active":"2024-03-31T15:53:53.528719+0000","last_peered":"2024-03-31T15:53:53.528719+0000","last_clean":"2024-03-31T15:53:53.528719+0000","last_became_active":"2024-03-31T15:53:46.757028+0000","last_became_peered":"2024-03-31T15:53:46.757028+0000","last_unstale":"2024-03-31T15:53:53.528719+0000","last_undegraded":"2024-03-31T15:53:53.528719+0000","last_fullsized":"2024-03-31T15:53:53.528719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:19.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306461+0000","last_change":"2024-03-31T15:53:51.799696+0000","last_active":"2024-03-31T15:53:53.306461+0000","last_peered":"2024-03-31T15:53:53.306461+0000","last_clean":"2024-03-31T15:53:53.306461+0000","last_became_active":"2024-03-31T15:53:51.799041+0000","last_became_peered":"2024-03-31T15:53:51.799041+0000","last_unstale":"2024-03-31T15:53:53.306461+0000","last_undegraded":"2024-03-31T15:53:53.306461+0000","last_fullsized":"2024-03-31T15:53:53.306461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789786+0000","last_change":"2024-03-31T15:53:47.767189+0000","last_active":"2024-03-31T15:53:52.789786+0000","last_peered":"2024-03-31T15:53:52.789786+0000","last_clean":"2024-03-31T15:53:52.789786+0000","last_became_active":"2024-03-31T15:53:47.766941+0000","last_became_peered":"2024-03-31T15:53:47.766941+0000","last_unstale":"2024-03-31T15:53:52.789786+0000","last_undegraded":"2024-03-31T15:53:52.789786+0000","last_fullsized":"2024-03-31T15:53:52.789786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:56.064763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921010+0000","last_change":"2024-03-31T15:53:46.755784+0000","last_active":"2024-03-31T15:53:52.921010+0000","last_peered":"2024-03-31T15:53:52.921010+0000","last_clean":"2024-03-31T15:53:52.921010+0000","last_became_active":"2024-03-31T15:53:46.755518+0000","last_became_peered":"2024-03-31T15:53:46.755518+0000","last_unstale":"2024-03-31T15:53:52.921010+0000","last_undegraded":"2024-03-31T15:53:52.921010+0000","last_fullsized":"2024-03-31T15:53:52.921010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:03.068278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359266+0000","last_change":"2024-03-31T15:53:51.805030+0000","last_active":"2024-03-31T15:53:53.359266+0000","last_peered":"2024-03-31T15:53:53.359266+0000","last_clean":"2024-03-31T15:53:53.359266+0000","last_became_active":"2024-03-31T15:53:51.804608+0000","last_became_peered":"2024-03-31T15:53:51.804608+0000","last_unstale":"2024-03-31T15:53:53.359266+0000","last_undegraded":"2024-03-31T15:53:53.359266+0000","last_fullsized":"2024-03-31T15:53:53.359266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792768+0000","last_change":"2024-03-31T15:53:47.763067+0000","last_active":"2024-03-31T15:53:52.792768+0000","last_peered":"2024-03-31T15:53:52.792768+0000","last_clean":"2024-03-31T15:53:52.792768+0000","last_became_active":"2024-03-31T15:53:47.762801+0000","last_became_peered":"2024-03-31T15:53:47.762801+0000","last_unstale":"2024-03-31T15:53:52.792768+0000","last_undegraded":"2024-03-31T15:53:52.792768+0000","last_fullsized":"2024-03-31T15:53:52.792768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:34.931240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528883+0000","last_change":"2024-03-31T15:53:46.763307+0000","last_active":"2024-03-31T15:53:53.528883+0000","last_peered":"2024-03-31T15:53:53.528883+0000","last_clean":"2024-03-31T15:53:53.528883+0000","last_became_active":"2024-03-31T15:53:46.763083+0000","last_became_peered":"2024-03-31T15:53:46.763083+0000","last_unstale":"2024-03-31T15:53:53.528883+0000","last_undegraded":"2024-03-31T15:53:53.528883+0000","last_fullsized":"2024-03-31T15:53:53.528883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:35:35.431822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789351+0000","last_change":"2024-03-31T15:53:51.792009+0000","last_active":"2024-03-31T15:53:52.789351+0000","last_peered":"2024-03-31T15:53:52.789351+0000","last_clean":"2024-03-31T15:53:52.789351+0000","last_became_active":"2024-03-31T15:53:51.791660+0000","last_became_peered":"2024-03-31T15:53:51.791660+0000","last_unstale":"2024-03-31T15:53:52.789351+0000","last_undegraded":"2024-03-31T15:53:52.789351+0000","last_fullsized":"2024-03-31T15:53:52.789351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:17.783876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792256+0000","last_change":"2024-03-31T15:53:47.771025+0000","last_active":"2024-03-31T15:53:52.792256+0000","last_peered":"2024-03-31T15:53:52.792256+0000","last_clean":"2024-03-31T15:53:52.792256+0000","last_became_active":"2024-03-31T15:53:47.769254+0000","last_became_peered":"2024-03-31T15:53:47.769254+0000","last_unstale":"2024-03-31T15:53:52.792256+0000","last_undegraded":"2024-03-31T15:53:52.792256+0000","last_fullsized":"2024-03-31T15:53:52.792256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:01.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793608+0000","last_change":"2024-03-31T15:53:46.759316+0000","last_active":"2024-03-31T15:53:52.793608+0000","last_peered":"2024-03-31T15:53:52.793608+0000","last_clean":"2024-03-31T15:53:52.793608+0000","last_became_active":"2024-03-31T15:53:46.758973+0000","last_became_peered":"2024-03-31T15:53:46.758973+0000","last_unstale":"2024-03-31T15:53:52.793608+0000","last_undegraded":"2024-03-31T15:53:52.793608+0000","last_fullsized":"2024-03-31T15:53:52.793608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:39.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792641+0000","last_change":"2024-03-31T15:53:51.795201+0000","last_active":"2024-03-31T15:53:52.792641+0000","last_peered":"2024-03-31T15:53:52.792641+0000","last_clean":"2024-03-31T15:53:52.792641+0000","last_became_active":"2024-03-31T15:53:51.794775+0000","last_became_peered":"2024-03-31T15:53:51.794775+0000","last_unstale":"2024-03-31T15:53:52.792641+0000","last_undegraded":"2024-03-31T15:53:52.792641+0000","last_fullsized":"2024-03-31T15:53:52.792641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:11.574086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791547+0000","last_change":"2024-03-31T15:53:47.760294+0000","last_active":"2024-03-31T15:53:52.791547+0000","last_peered":"2024-03-31T15:53:52.791547+0000","last_clean":"2024-03-31T15:53:52.791547+0000","last_became_active":"2024-03-31T15:53:47.760011+0000","last_became_peered":"2024-03-31T15:53:47.760011+0000","last_unstale":"2024-03-31T15:53:52.791547+0000","last_undegraded":"2024-03-31T15:53:52.791547+0000","last_fullsized":"2024-03-31T15:53:52.791547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:11.819602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528298+0000","last_change":"2024-03-31T15:53:46.759221+0000","last_active":"2024-03-31T15:53:53.528298+0000","last_peered":"2024-03-31T15:53:53.528298+0000","last_clean":"2024-03-31T15:53:53.528298+0000","last_became_active":"2024-03-31T15:53:46.758926+0000","last_became_peered":"2024-03-31T15:53:46.758926+0000","last_unstale":"2024-03-31T15:53:53.528298+0000","last_undegraded":"2024-03-31T15:53:53.528298+0000","last_fullsized":"2024-03-31T15:53:53.528298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359571+0000","last_change":"2024-03-31T15:53:51.794069+0000","last_active":"2024-03-31T15:53:53.359571+0000","last_peered":"2024-03-31T15:53:53.359571+0000","last_clean":"2024-03-31T15:53:53.359571+0000","last_became_active":"2024-03-31T15:53:51.793665+0000","last_became_peered":"2024-03-31T15:53:51.793665+0000","last_unstale":"2024-03-31T15:53:53.359571+0000","last_undegraded":"2024-03-31T15:53:53.359571+0000","last_fullsized":"2024-03-31T15:53:53.359571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306482+0000","last_change":"2024-03-31T15:53:47.763992+0000","last_active":"2024-03-31T15:53:53.306482+0000","last_peered":"2024-03-31T15:53:53.306482+0000","last_clean":"2024-03-31T15:53:53.306482+0000","last_became_active":"2024-03-31T15:53:47.763743+0000","last_became_peered":"2024-03-31T15:53:47.763743+0000","last_unstale":"2024-03-31T15:53:53.306482+0000","last_undegraded":"2024-03-31T15:53:53.306482+0000","last_fullsized":"2024-03-31T15:53:53.306482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306483+0000","last_change":"2024-03-31T15:53:46.764342+0000","last_active":"2024-03-31T15:53:53.306483+0000","last_peered":"2024-03-31T15:53:53.306483+0000","last_clean":"2024-03-31T15:53:53.306483+0000","last_became_active":"2024-03-31T15:53:46.763064+0000","last_became_peered":"2024-03-31T15:53:46.763064+0000","last_unstale":"2024-03-31T15:53:53.306483+0000","last_undegraded":"2024-03-31T15:53:53.306483+0000","last_fullsized":"2024-03-31T15:53:53.306483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248662+0000","last_change":"2024-03-31T15:53:51.802982+0000","last_active":"2024-03-31T15:53:53.248662+0000","last_peered":"2024-03-31T15:53:53.248662+0000","last_clean":"2024-03-31T15:53:53.248662+0000","last_became_active":"2024-03-31T15:53:51.802619+0000","last_became_peered":"2024-03-31T15:53:51.802619+0000","last_unstale":"2024-03-31T15:53:53.248662+0000","last_undegraded":"2024-03-31T15:53:53.248662+0000","last_fullsized":"2024-03-31T15:53:53.248662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:12:18.187743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791793+0000","last_change":"2024-03-31T15:53:47.763046+0000","last_active":"2024-03-31T15:53:52.791793+0000","last_peered":"2024-03-31T15:53:52.791793+0000","last_clean":"2024-03-31T15:53:52.791793+0000","last_became_active":"2024-03-31T15:53:47.762791+0000","last_became_peered":"2024-03-31T15:53:47.762791+0000","last_unstale":"2024-03-31T15:53:52.791793+0000","last_undegraded":"2024-03-31T15:53:52.791793+0000","last_fullsized":"2024-03-31T15:53:52.791793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:32:01.910382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790646+0000","last_change":"2024-03-31T15:53:46.765124+0000","last_active":"2024-03-31T15:53:52.790646+0000","last_peered":"2024-03-31T15:53:52.790646+0000","last_clean":"2024-03-31T15:53:52.790646+0000","last_became_active":"2024-03-31T15:53:46.764856+0000","last_became_peered":"2024-03-31T15:53:46.764856+0000","last_unstale":"2024-03-31T15:53:52.790646+0000","last_undegraded":"2024-03-31T15:53:52.790646+0000","last_fullsized":"2024-03-31T15:53:52.790646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248292+0000","last_change":"2024-03-31T15:53:51.807271+0000","last_active":"2024-03-31T15:53:53.248292+0000","last_peered":"2024-03-31T15:53:53.248292+0000","last_clean":"2024-03-31T15:53:53.248292+0000","last_became_active":"2024-03-31T15:53:51.806909+0000","last_became_peered":"2024-03-31T15:53:51.806909+0000","last_unstale":"2024-03-31T15:53:53.248292+0000","last_undegraded":"2024-03-31T15:53:53.248292+0000","last_fullsized":"2024-03-31T15:53:53.248292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:14:12.015472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359081+0000","last_change":"2024-03-31T15:53:47.762410+0000","last_active":"2024-03-31T15:53:53.359081+0000","last_peered":"2024-03-31T15:53:53.359081+0000","last_clean":"2024-03-31T15:53:53.359081+0000","last_became_active":"2024-03-31T15:53:47.762145+0000","last_became_peered":"2024-03-31T15:53:47.762145+0000","last_unstale":"2024-03-31T15:53:53.359081+0000","last_undegraded":"2024-03-31T15:53:53.359081+0000","last_fullsized":"2024-03-31T15:53:53.359081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359350+0000","last_change":"2024-03-31T15:53:46.756308+0000","last_active":"2024-03-31T15:53:53.359350+0000","last_peered":"2024-03-31T15:53:53.359350+0000","last_clean":"2024-03-31T15:53:53.359350+0000","last_became_active":"2024-03-31T15:53:46.756010+0000","last_became_peered":"2024-03-31T15:53:46.756010+0000","last_unstale":"2024-03-31T15:53:53.359350+0000","last_undegraded":"2024-03-31T15:53:53.359350+0000","last_fullsized":"2024-03-31T15:53:53.359350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:30:06.680148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790084+0000","last_change":"2024-03-31T15:53:51.806516+0000","last_active":"2024-03-31T15:53:52.790084+0000","last_peered":"2024-03-31T15:53:52.790084+0000","last_clean":"2024-03-31T15:53:52.790084+0000","last_became_active":"2024-03-31T15:53:51.805985+0000","last_became_peered":"2024-03-31T15:53:51.805985+0000","last_unstale":"2024-03-31T15:53:52.790084+0000","last_undegraded":"2024-03-31T15:53:52.790084+0000","last_fullsized":"2024-03-31T15:53:52.790084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:12:27.917062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791963+0000","last_change":"2024-03-31T15:53:47.768000+0000","last_active":"2024-03-31T15:53:52.791963+0000","last_peered":"2024-03-31T15:53:52.791963+0000","last_clean":"2024-03-31T15:53:52.791963+0000","last_became_active":"2024-03-31T15:53:47.767710+0000","last_became_peered":"2024-03-31T15:53:47.767710+0000","last_unstale":"2024-03-31T15:53:52.791963+0000","last_undegraded":"2024-03-31T15:53:52.791963+0000","last_fullsized":"2024-03-31T15:53:52.791963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:35.607484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:54:02.857882+0000","last_change":"2024-03-31T15:53:46.764364+0000","last_active":"2024-03-31T15:54:02.857882+0000","last_peered":"2024-03-31T15:54:02.857882+0000","last_clean":"2024-03-31T15:54:02.857882+0000","last_became_active":"2024-03-31T15:53:46.763234+0000","last_became_peered":"2024-03-31T15:53:46.763234+0000","last_unstale":"2024-03-31T15:54:02.857882+0000","last_undegraded":"2024-03-31T15:54:02.857882+0000","last_fullsized":"2024-03-31T15:54:02.857882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920563+0000","last_change":"2024-03-31T15:53:51.794393+0000","last_active":"2024-03-31T15:53:52.920563+0000","last_peered":"2024-03-31T15:53:52.920563+0000","last_clean":"2024-03-31T15:53:52.920563+0000","last_became_active":"2024-03-31T15:53:51.793984+0000","last_became_peered":"2024-03-31T15:53:51.793984+0000","last_unstale":"2024-03-31T15:53:52.920563+0000","last_undegraded":"2024-03-31T15:53:52.920563+0000","last_fullsized":"2024-03-31T15:53:52.920563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793726+0000","last_change":"2024-03-31T15:53:47.759150+0000","last_active":"2024-03-31T15:53:52.793726+0000","last_peered":"2024-03-31T15:53:52.793726+0000","last_clean":"2024-03-31T15:53:52.793726+0000","last_became_active":"2024-03-31T15:53:47.757846+0000","last_became_peered":"2024-03-31T15:53:47.757846+0000","last_unstale":"2024-03-31T15:53:52.793726+0000","last_undegraded":"2024-03-31T15:53:52.793726+0000","last_fullsized":"2024-03-31T15:53:52.793726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:16.643130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920932+0000","last_change":"2024-03-31T15:53:46.763645+0000","last_active":"2024-03-31T15:53:52.920932+0000","last_peered":"2024-03-31T15:53:52.920932+0000","last_clean":"2024-03-31T15:53:52.920932+0000","last_became_active":"2024-03-31T15:53:46.763370+0000","last_became_peered":"2024-03-31T15:53:46.763370+0000","last_unstale":"2024-03-31T15:53:52.920932+0000","last_undegraded":"2024-03-31T15:53:52.920932+0000","last_fullsized":"2024-03-31T15:53:52.920932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:15.635255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793545+0000","last_change":"2024-03-31T15:53:51.804406+0000","last_active":"2024-03-31T15:53:52.793545+0000","last_peered":"2024-03-31T15:53:52.793545+0000","last_clean":"2024-03-31T15:53:52.793545+0000","last_became_active":"2024-03-31T15:53:51.803887+0000","last_became_peered":"2024-03-31T15:53:51.803887+0000","last_unstale":"2024-03-31T15:53:52.793545+0000","last_undegraded":"2024-03-31T15:53:52.793545+0000","last_fullsized":"2024-03-31T15:53:52.793545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359778+0000","last_change":"2024-03-31T15:53:47.771467+0000","last_active":"2024-03-31T15:53:53.359778+0000","last_peered":"2024-03-31T15:53:53.359778+0000","last_clean":"2024-03-31T15:53:53.359778+0000","last_became_active":"2024-03-31T15:53:47.771267+0000","last_became_peered":"2024-03-31T15:53:47.771267+0000","last_unstale":"2024-03-31T15:53:53.359778+0000","last_undegraded":"2024-03-31T15:53:53.359778+0000","last_fullsized":"2024-03-31T15:53:53.359778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790236+0000","last_change":"2024-03-31T15:53:46.758836+0000","last_active":"2024-03-31T15:53:52.790236+0000","last_peered":"2024-03-31T15:53:52.790236+0000","last_clean":"2024-03-31T15:53:52.790236+0000","last_became_active":"2024-03-31T15:53:46.758359+0000","last_became_peered":"2024-03-31T15:53:46.758359+0000","last_unstale":"2024-03-31T15:53:52.790236+0000","last_undegraded":"2024-03-31T15:53:52.790236+0000","last_fullsized":"2024-03-31T15:53:52.790236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247843+0000","last_change":"2024-03-31T15:53:51.800437+0000","last_active":"2024-03-31T15:53:53.247843+0000","last_peered":"2024-03-31T15:53:53.247843+0000","last_clean":"2024-03-31T15:53:53.247843+0000","last_became_active":"2024-03-31T15:53:51.799728+0000","last_became_peered":"2024-03-31T15:53:51.799728+0000","last_unstale":"2024-03-31T15:53:53.247843+0000","last_undegraded":"2024-03-31T15:53:53.247843+0000","last_fullsized":"2024-03-31T15:53:53.247843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:45.272362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790191+0000","last_change":"2024-03-31T15:53:47.765037+0000","last_active":"2024-03-31T15:53:52.790191+0000","last_peered":"2024-03-31T15:53:52.790191+0000","last_clean":"2024-03-31T15:53:52.790191+0000","last_became_active":"2024-03-31T15:53:47.764764+0000","last_became_peered":"2024-03-31T15:53:47.764764+0000","last_unstale":"2024-03-31T15:53:52.790191+0000","last_undegraded":"2024-03-31T15:53:52.790191+0000","last_fullsized":"2024-03-31T15:53:52.790191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:07.287524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307175+0000","last_change":"2024-03-31T15:53:46.765922+0000","last_active":"2024-03-31T15:53:53.307175+0000","last_peered":"2024-03-31T15:53:53.307175+0000","last_clean":"2024-03-31T15:53:53.307175+0000","last_became_active":"2024-03-31T15:53:46.765530+0000","last_became_peered":"2024-03-31T15:53:46.765530+0000","last_unstale":"2024-03-31T15:53:53.307175+0000","last_undegraded":"2024-03-31T15:53:53.307175+0000","last_fullsized":"2024-03-31T15:53:53.307175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247784+0000","last_change":"2024-03-31T15:53:51.803591+0000","last_active":"2024-03-31T15:53:53.247784+0000","last_peered":"2024-03-31T15:53:53.247784+0000","last_clean":"2024-03-31T15:53:53.247784+0000","last_became_active":"2024-03-31T15:53:51.803139+0000","last_became_peered":"2024-03-31T15:53:51.803139+0000","last_unstale":"2024-03-31T15:53:53.247784+0000","last_undegraded":"2024-03-31T15:53:53.247784+0000","last_fullsized":"2024-03-31T15:53:53.247784+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:08.680601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359558+0000","last_change":"2024-03-31T15:53:51.809399+0000","last_active":"2024-03-31T15:53:53.359558+0000","last_peered":"2024-03-31T15:53:53.359558+0000","last_clean":"2024-03-31T15:53:53.359558+0000","last_became_active":"2024-03-31T15:53:51.808396+0000","last_became_peered":"2024-03-31T15:53:51.808396+0000","last_unstale":"2024-03-31T15:53:53.359558+0000","last_undegraded":"2024-03-31T15:53:53.359558+0000","last_fullsized":"2024-03-31T15:53:53.359558+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920147+0000","last_change":"2024-03-31T15:53:47.770618+0000","last_active":"2024-03-31T15:53:52.920147+0000","last_peered":"2024-03-31T15:53:52.920147+0000","last_clean":"2024-03-31T15:53:52.920147+0000","last_became_active":"2024-03-31T15:53:47.770333+0000","last_became_peered":"2024-03-31T15:53:47.770333+0000","last_unstale":"2024-03-31T15:53:52.920147+0000","last_undegraded":"2024-03-31T15:53:52.920147+0000","last_fullsized":"2024-03-31T15:53:52.920147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789491+0000","last_change":"2024-03-31T15:53:46.759181+0000","last_active":"2024-03-31T15:53:52.789491+0000","last_peered":"2024-03-31T15:53:52.789491+0000","last_clean":"2024-03-31T15:53:52.789491+0000","last_became_active":"2024-03-31T15:53:46.758355+0000","last_became_peered":"2024-03-31T15:53:46.758355+0000","last_unstale":"2024-03-31T15:53:52.789491+0000","last_undegraded":"2024-03-31T15:53:52.789491+0000","last_fullsized":"2024-03-31T15:53:52.789491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306986+0000","last_change":"2024-03-31T15:53:51.796763+0000","last_active":"2024-03-31T15:53:53.306986+0000","last_peered":"2024-03-31T15:53:53.306986+0000","last_clean":"2024-03-31T15:53:53.306986+0000","last_became_active":"2024-03-31T15:53:51.796351+0000","last_became_peered":"2024-03-31T15:53:51.796351+0000","last_unstale":"2024-03-31T15:53:53.306986+0000","last_undegraded":"2024-03-31T15:53:53.306986+0000","last_fullsized":"2024-03-31T15:53:53.306986+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247797+0000","last_change":"2024-03-31T15:53:47.766443+0000","last_active":"2024-03-31T15:53:53.247797+0000","last_peered":"2024-03-31T15:53:53.247797+0000","last_clean":"2024-03-31T15:53:53.247797+0000","last_became_active":"2024-03-31T15:53:47.766107+0000","last_became_peered":"2024-03-31T15:53:47.766107+0000","last_unstale":"2024-03-31T15:53:53.247797+0000","last_undegraded":"2024-03-31T15:53:53.247797+0000","last_fullsized":"2024-03-31T15:53:53.247797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:50:47.205172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248000+0000","last_change":"2024-03-31T15:53:46.764698+0000","last_active":"2024-03-31T15:53:53.248000+0000","last_peered":"2024-03-31T15:53:53.248000+0000","last_clean":"2024-03-31T15:53:53.248000+0000","last_became_active":"2024-03-31T15:53:46.764345+0000","last_became_peered":"2024-03-31T15:53:46.764345+0000","last_unstale":"2024-03-31T15:53:53.248000+0000","last_undegraded":"2024-03-31T15:53:53.248000+0000","last_fullsized":"2024-03-31T15:53:53.248000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:42.990113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306648+0000","last_change":"2024-03-31T15:53:46.765106+0000","last_active":"2024-03-31T15:53:53.306648+0000","last_peered":"2024-03-31T15:53:53.306648+0000","last_clean":"2024-03-31T15:53:53.306648+0000","last_became_active":"2024-03-31T15:53:46.764281+0000","last_became_peered":"2024-03-31T15:53:46.764281+0000","last_unstale":"2024-03-31T15:53:53.306648+0000","last_undegraded":"2024-03-31T15:53:53.306648+0000","last_fullsized":"2024-03-31T15:53:53.306648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359310+0000","last_change":"2024-03-31T15:53:47.757501+0000","last_active":"2024-03-31T15:53:53.359310+0000","last_peered":"2024-03-31T15:53:53.359310+0000","last_clean":"2024-03-31T15:53:53.359310+0000","last_became_active":"2024-03-31T15:53:47.757160+0000","last_became_peered":"2024-03-31T15:53:47.757160+0000","last_unstale":"2024-03-31T15:53:53.359310+0000","last_undegraded":"2024-03-31T15:53:53.359310+0000","last_fullsized":"2024-03-31T15:53:53.359310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528662+0000","last_change":"2024-03-31T15:53:51.794610+0000","last_active":"2024-03-31T15:53:53.528662+0000","last_peered":"2024-03-31T15:53:53.528662+0000","last_clean":"2024-03-31T15:53:53.528662+0000","last_became_active":"2024-03-31T15:53:51.794228+0000","last_became_peered":"2024-03-31T15:53:51.794228+0000","last_unstale":"2024-03-31T15:53:53.528662+0000","last_undegraded":"2024-03-31T15:53:53.528662+0000","last_fullsized":"2024-03-31T15:53:53.528662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:55.277607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248028+0000","last_change":"2024-03-31T15:53:46.762539+0000","last_active":"2024-03-31T15:53:53.248028+0000","last_peered":"2024-03-31T15:53:53.248028+0000","last_clean":"2024-03-31T15:53:53.248028+0000","last_became_active":"2024-03-31T15:53:46.761475+0000","last_became_peered":"2024-03-31T15:53:46.761475+0000","last_unstale":"2024-03-31T15:53:53.248028+0000","last_undegraded":"2024-03-31T15:53:53.248028+0000","last_fullsized":"2024-03-31T15:53:53.248028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:36:29.320997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793114+0000","last_change":"2024-03-31T15:53:47.758390+0000","last_active":"2024-03-31T15:53:52.793114+0000","last_peered":"2024-03-31T15:53:52.793114+0000","last_clean":"2024-03-31T15:53:52.793114+0000","last_became_active":"2024-03-31T15:53:47.754114+0000","last_became_peered":"2024-03-31T15:53:47.754114+0000","last_unstale":"2024-03-31T15:53:52.793114+0000","last_undegraded":"2024-03-31T15:53:52.793114+0000","last_fullsized":"2024-03-31T15:53:52.793114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:12.842845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248390+0000","last_change":"2024-03-31T15:53:51.799618+0000","last_active":"2024-03-31T15:53:53.248390+0000","last_peered":"2024-03-31T15:53:53.248390+0000","last_clean":"2024-03-31T15:53:53.248390+0000","last_became_active":"2024-03-31T15:53:51.799171+0000","last_became_peered":"2024-03-31T15:53:51.799171+0000","last_unstale":"2024-03-31T15:53:53.248390+0000","last_undegraded":"2024-03-31T15:53:53.248390+0000","last_fullsized":"2024-03-31T15:53:53.248390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:48.155512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306339+0000","last_change":"2024-03-31T15:53:47.765092+0000","last_active":"2024-03-31T15:53:53.306339+0000","last_peered":"2024-03-31T15:53:53.306339+0000","last_clean":"2024-03-31T15:53:53.306339+0000","last_became_active":"2024-03-31T15:53:47.764701+0000","last_became_peered":"2024-03-31T15:53:47.764701+0000","last_unstale":"2024-03-31T15:53:53.306339+0000","last_undegraded":"2024-03-31T15:53:53.306339+0000","last_fullsized":"2024-03-31T15:53:53.306339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528367+0000","last_change":"2024-03-31T15:53:47.765373+0000","last_active":"2024-03-31T15:53:53.528367+0000","last_peered":"2024-03-31T15:53:53.528367+0000","last_clean":"2024-03-31T15:53:53.528367+0000","last_became_active":"2024-03-31T15:53:47.764321+0000","last_became_peered":"2024-03-31T15:53:47.764321+0000","last_unstale":"2024-03-31T15:53:53.528367+0000","last_undegraded":"2024-03-31T15:53:53.528367+0000","last_fullsized":"2024-03-31T15:53:53.528367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:19.777139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790598+0000","last_change":"2024-03-31T15:53:47.762666+0000","last_active":"2024-03-31T15:53:52.790598+0000","last_peered":"2024-03-31T15:53:52.790598+0000","last_clean":"2024-03-31T15:53:52.790598+0000","last_became_active":"2024-03-31T15:53:47.762284+0000","last_became_peered":"2024-03-31T15:53:47.762284+0000","last_unstale":"2024-03-31T15:53:52.790598+0000","last_undegraded":"2024-03-31T15:53:52.790598+0000","last_fullsized":"2024-03-31T15:53:52.790598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:44.058276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359256+0000","last_change":"2024-03-31T15:53:47.762200+0000","last_active":"2024-03-31T15:53:53.359256+0000","last_peered":"2024-03-31T15:53:53.359256+0000","last_clean":"2024-03-31T15:53:53.359256+0000","last_became_active":"2024-03-31T15:53:47.761993+0000","last_became_peered":"2024-03-31T15:53:47.761993+0000","last_unstale":"2024-03-31T15:53:53.359256+0000","last_undegraded":"2024-03-31T15:53:53.359256+0000","last_fullsized":"2024-03-31T15:53:53.359256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248436+0000","last_change":"2024-03-31T15:53:47.758308+0000","last_active":"2024-03-31T15:53:53.248436+0000","last_peered":"2024-03-31T15:53:53.248436+0000","last_clean":"2024-03-31T15:53:53.248436+0000","last_became_active":"2024-03-31T15:53:47.757996+0000","last_became_peered":"2024-03-31T15:53:47.757996+0000","last_unstale":"2024-03-31T15:53:53.248436+0000","last_undegraded":"2024-03-31T15:53:53.248436+0000","last_fullsized":"2024-03-31T15:53:53.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:24:07.611521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920834+0000","last_change":"2024-03-31T15:53:47.754477+0000","last_active":"2024-03-31T15:53:52.920834+0000","last_peered":"2024-03-31T15:53:52.920834+0000","last_clean":"2024-03-31T15:53:52.920834+0000","last_became_active":"2024-03-31T15:53:47.754071+0000","last_became_peered":"2024-03-31T15:53:47.754071+0000","last_unstale":"2024-03-31T15:53:52.920834+0000","last_undegraded":"2024-03-31T15:53:52.920834+0000","last_fullsized":"2024-03-31T15:53:52.920834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307045+0000","last_change":"2024-03-31T15:53:47.761169+0000","last_active":"2024-03-31T15:53:53.307045+0000","last_peered":"2024-03-31T15:53:53.307045+0000","last_clean":"2024-03-31T15:53:53.307045+0000","last_became_active":"2024-03-31T15:53:47.760902+0000","last_became_peered":"2024-03-31T15:53:47.760902+0000","last_unstale":"2024-03-31T15:53:53.307045+0000","last_undegraded":"2024-03-31T15:53:53.307045+0000","last_fullsized":"2024-03-31T15:53:53.307045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248647+0000","last_change":"2024-03-31T15:53:47.758495+0000","last_active":"2024-03-31T15:53:53.248647+0000","last_peered":"2024-03-31T15:53:53.248647+0000","last_clean":"2024-03-31T15:53:53.248647+0000","last_became_active":"2024-03-31T15:53:47.758146+0000","last_became_peered":"2024-03-31T15:53:47.758146+0000","last_unstale":"2024-03-31T15:53:53.248647+0000","last_undegraded":"2024-03-31T15:53:53.248647+0000","last_fullsized":"2024-03-31T15:53:53.248647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:40:45.579157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789642+0000","last_change":"2024-03-31T15:53:47.759644+0000","last_active":"2024-03-31T15:53:52.789642+0000","last_peered":"2024-03-31T15:53:52.789642+0000","last_clean":"2024-03-31T15:53:52.789642+0000","last_became_active":"2024-03-31T15:53:47.759278+0000","last_became_peered":"2024-03-31T15:53:47.759278+0000","last_unstale":"2024-03-31T15:53:52.789642+0000","last_undegraded":"2024-03-31T15:53:52.789642+0000","last_fullsized":"2024-03-31T15:53:52.789642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:36.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247837+0000","last_change":"2024-03-31T15:53:47.759026+0000","last_active":"2024-03-31T15:53:53.247837+0000","last_peered":"2024-03-31T15:53:53.247837+0000","last_clean":"2024-03-31T15:53:53.247837+0000","last_became_active":"2024-03-31T15:53:47.758752+0000","last_became_peered":"2024-03-31T15:53:47.758752+0000","last_unstale":"2024-03-31T15:53:53.247837+0000","last_undegraded":"2024-03-31T15:53:53.247837+0000","last_fullsized":"2024-03-31T15:53:53.247837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:05:25.701168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790550+0000","last_change":"2024-03-31T15:53:47.762998+0000","last_active":"2024-03-31T15:53:52.790550+0000","last_peered":"2024-03-31T15:53:52.790550+0000","last_clean":"2024-03-31T15:53:52.790550+0000","last_became_active":"2024-03-31T15:53:47.762724+0000","last_became_peered":"2024-03-31T15:53:47.762724+0000","last_unstale":"2024-03-31T15:53:52.790550+0000","last_undegraded":"2024-03-31T15:53:52.790550+0000","last_fullsized":"2024-03-31T15:53:52.790550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359161+0000","last_change":"2024-03-31T15:53:47.761675+0000","last_active":"2024-03-31T15:53:53.359161+0000","last_peered":"2024-03-31T15:53:53.359161+0000","last_clean":"2024-03-31T15:53:53.359161+0000","last_became_active":"2024-03-31T15:53:47.757931+0000","last_became_peered":"2024-03-31T15:53:47.757931+0000","last_unstale":"2024-03-31T15:53:53.359161+0000","last_undegraded":"2024-03-31T15:53:53.359161+0000","last_fullsized":"2024-03-31T15:53:53.359161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:15.515856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306768+0000","last_change":"2024-03-31T15:53:47.770705+0000","last_active":"2024-03-31T15:53:53.306768+0000","last_peered":"2024-03-31T15:53:53.306768+0000","last_clean":"2024-03-31T15:53:53.306768+0000","last_became_active":"2024-03-31T15:53:47.770485+0000","last_became_peered":"2024-03-31T15:53:47.770485+0000","last_unstale":"2024-03-31T15:53:53.306768+0000","last_undegraded":"2024-03-31T15:53:53.306768+0000","last_fullsized":"2024-03-31T15:53:53.306768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792240+0000","last_change":"2024-03-31T15:53:47.761166+0000","last_active":"2024-03-31T15:53:52.792240+0000","last_peered":"2024-03-31T15:53:52.792240+0000","last_clean":"2024-03-31T15:53:52.792240+0000","last_became_active":"2024-03-31T15:53:47.760825+0000","last_became_peered":"2024-03-31T15:53:47.760825+0000","last_unstale":"2024-03-31T15:53:52.792240+0000","last_undegraded":"2024-03-31T15:53:52.792240+0000","last_fullsized":"2024-03-31T15:53:52.792240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:56.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528829+0000","last_change":"2024-03-31T15:53:47.765006+0000","last_active":"2024-03-31T15:53:53.528829+0000","last_peered":"2024-03-31T15:53:53.528829+0000","last_clean":"2024-03-31T15:53:53.528829+0000","last_became_active":"2024-03-31T15:53:47.760192+0000","last_became_peered":"2024-03-31T15:53:47.760192+0000","last_unstale":"2024-03-31T15:53:53.528829+0000","last_undegraded":"2024-03-31T15:53:53.528829+0000","last_fullsized":"2024-03-31T15:53:53.528829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:26.202073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790075+0000","last_change":"2024-03-31T15:53:47.765871+0000","last_active":"2024-03-31T15:53:52.790075+0000","last_peered":"2024-03-31T15:53:52.790075+0000","last_clean":"2024-03-31T15:53:52.790075+0000","last_became_active":"2024-03-31T15:53:47.764257+0000","last_became_peered":"2024-03-31T15:53:47.764257+0000","last_unstale":"2024-03-31T15:53:52.790075+0000","last_undegraded":"2024-03-31T15:53:52.790075+0000","last_fullsized":"2024-03-31T15:53:52.790075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:40.058921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789357+0000","last_change":"2024-03-31T15:53:47.764320+0000","last_active":"2024-03-31T15:53:52.789357+0000","last_peered":"2024-03-31T15:53:52.789357+0000","last_clean":"2024-03-31T15:53:52.789357+0000","last_became_active":"2024-03-31T15:53:47.764067+0000","last_became_peered":"2024-03-31T15:53:47.764067+0000","last_unstale":"2024-03-31T15:53:52.789357+0000","last_undegraded":"2024-03-31T15:53:52.789357+0000","last_fullsized":"2024-03-31T15:53:52.789357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920242+0000","last_change":"2024-03-31T15:53:47.755404+0000","last_active":"2024-03-31T15:53:52.920242+0000","last_peered":"2024-03-31T15:53:52.920242+0000","last_clean":"2024-03-31T15:53:52.920242+0000","last_became_active":"2024-03-31T15:53:47.755133+0000","last_became_peered":"2024-03-31T15:53:47.755133+0000","last_unstale":"2024-03-31T15:53:52.920242+0000","last_undegraded":"2024-03-31T15:53:52.920242+0000","last_fullsized":"2024-03-31T15:53:52.920242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359051+0000","last_change":"2024-03-31T15:53:47.751494+0000","last_active":"2024-03-31T15:53:53.359051+0000","last_peered":"2024-03-31T15:53:53.359051+0000","last_clean":"2024-03-31T15:53:53.359051+0000","last_became_active":"2024-03-31T15:53:47.751219+0000","last_became_peered":"2024-03-31T15:53:47.751219+0000","last_unstale":"2024-03-31T15:53:53.359051+0000","last_undegraded":"2024-03-31T15:53:53.359051+0000","last_fullsized":"2024-03-31T15:53:53.359051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247881+0000","last_change":"2024-03-31T15:53:47.766595+0000","last_active":"2024-03-31T15:53:53.247881+0000","last_peered":"2024-03-31T15:53:53.247881+0000","last_clean":"2024-03-31T15:53:53.247881+0000","last_became_active":"2024-03-31T15:53:47.766209+0000","last_became_peered":"2024-03-31T15:53:47.766209+0000","last_unstale":"2024-03-31T15:53:53.247881+0000","last_undegraded":"2024-03-31T15:53:53.247881+0000","last_fullsized":"2024-03-31T15:53:53.247881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:51.207164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793406+0000","last_change":"2024-03-31T15:53:47.762468+0000","last_active":"2024-03-31T15:53:52.793406+0000","last_peered":"2024-03-31T15:53:52.793406+0000","last_clean":"2024-03-31T15:53:52.793406+0000","last_became_active":"2024-03-31T15:53:47.762140+0000","last_became_peered":"2024-03-31T15:53:47.762140+0000","last_unstale":"2024-03-31T15:53:52.793406+0000","last_undegraded":"2024-03-31T15:53:52.793406+0000","last_fullsized":"2024-03-31T15:53:52.793406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:04.076153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306895+0000","last_change":"2024-03-31T15:53:47.774515+0000","last_active":"2024-03-31T15:53:53.306895+0000","last_peered":"2024-03-31T15:53:53.306895+0000","last_clean":"2024-03-31T15:53:53.306895+0000","last_became_active":"2024-03-31T15:53:47.774203+0000","last_became_peered":"2024-03-31T15:53:47.774203+0000","last_unstale":"2024-03-31T15:53:53.306895+0000","last_undegraded":"2024-03-31T15:53:53.306895+0000","last_fullsized":"2024-03-31T15:53:53.306895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307016+0000","last_change":"2024-03-31T15:53:47.775151+0000","last_active":"2024-03-31T15:53:53.307016+0000","last_peered":"2024-03-31T15:53:53.307016+0000","last_clean":"2024-03-31T15:53:53.307016+0000","last_became_active":"2024-03-31T15:53:47.774661+0000","last_became_peered":"2024-03-31T15:53:47.774661+0000","last_unstale":"2024-03-31T15:53:53.307016+0000","last_undegraded":"2024-03-31T15:53:53.307016+0000","last_fullsized":"2024-03-31T15:53:53.307016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792705+0000","last_change":"2024-03-31T15:53:47.772389+0000","last_active":"2024-03-31T15:53:52.792705+0000","last_peered":"2024-03-31T15:53:52.792705+0000","last_clean":"2024-03-31T15:53:52.792705+0000","last_became_active":"2024-03-31T15:53:47.772131+0000","last_became_peered":"2024-03-31T15:53:47.772131+0000","last_unstale":"2024-03-31T15:53:52.792705+0000","last_undegraded":"2024-03-31T15:53:52.792705+0000","last_fullsized":"2024-03-31T15:53:52.792705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:35.224566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247670+0000","last_change":"2024-03-31T15:53:47.772995+0000","last_active":"2024-03-31T15:53:53.247670+0000","last_peered":"2024-03-31T15:53:53.247670+0000","last_clean":"2024-03-31T15:53:53.247670+0000","last_became_active":"2024-03-31T15:53:47.772708+0000","last_became_peered":"2024-03-31T15:53:47.772708+0000","last_unstale":"2024-03-31T15:53:53.247670+0000","last_undegraded":"2024-03-31T15:53:53.247670+0000","last_fullsized":"2024-03-31T15:53:53.247670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:12:55.991650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791467+0000","last_change":"2024-03-31T15:53:47.766440+0000","last_active":"2024-03-31T15:53:52.791467+0000","last_peered":"2024-03-31T15:53:52.791467+0000","last_clean":"2024-03-31T15:53:52.791467+0000","last_became_active":"2024-03-31T15:53:47.766187+0000","last_became_peered":"2024-03-31T15:53:47.766187+0000","last_unstale":"2024-03-31T15:53:52.791467+0000","last_undegraded":"2024-03-31T15:53:52.791467+0000","last_fullsized":"2024-03-31T15:53:52.791467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:53.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790499+0000","last_change":"2024-03-31T15:53:47.778379+0000","last_active":"2024-03-31T15:53:52.790499+0000","last_peered":"2024-03-31T15:53:52.790499+0000","last_clean":"2024-03-31T15:53:52.790499+0000","last_became_active":"2024-03-31T15:53:47.778049+0000","last_became_peered":"2024-03-31T15:53:47.778049+0000","last_unstale":"2024-03-31T15:53:52.790499+0000","last_undegraded":"2024-03-31T15:53:52.790499+0000","last_fullsized":"2024-03-31T15:53:52.790499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791659+0000","last_change":"2024-03-31T15:53:47.761184+0000","last_active":"2024-03-31T15:53:52.791659+0000","last_peered":"2024-03-31T15:53:52.791659+0000","last_clean":"2024-03-31T15:53:52.791659+0000","last_became_active":"2024-03-31T15:53:47.760863+0000","last_became_peered":"2024-03-31T15:53:47.760863+0000","last_unstale":"2024-03-31T15:53:52.791659+0000","last_undegraded":"2024-03-31T15:53:52.791659+0000","last_fullsized":"2024-03-31T15:53:52.791659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:08:38.336677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789652+0000","last_change":"2024-03-31T15:53:47.766801+0000","last_active":"2024-03-31T15:53:52.789652+0000","last_peered":"2024-03-31T15:53:52.789652+0000","last_clean":"2024-03-31T15:53:52.789652+0000","last_became_active":"2024-03-31T15:53:47.766512+0000","last_became_peered":"2024-03-31T15:53:47.766512+0000","last_unstale":"2024-03-31T15:53:52.789652+0000","last_undegraded":"2024-03-31T15:53:52.789652+0000","last_fullsized":"2024-03-31T15:53:52.789652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:25.798405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359475+0000","last_change":"2024-03-31T15:53:47.756732+0000","last_active":"2024-03-31T15:53:53.359475+0000","last_peered":"2024-03-31T15:53:53.359475+0000","last_clean":"2024-03-31T15:53:53.359475+0000","last_became_active":"2024-03-31T15:53:47.756466+0000","last_became_peered":"2024-03-31T15:53:47.756466+0000","last_unstale":"2024-03-31T15:53:53.359475+0000","last_undegraded":"2024-03-31T15:53:53.359475+0000","last_fullsized":"2024-03-31T15:53:53.359475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921162+0000","last_change":"2024-03-31T15:53:47.763716+0000","last_active":"2024-03-31T15:53:52.921162+0000","last_peered":"2024-03-31T15:53:52.921162+0000","last_clean":"2024-03-31T15:53:52.921162+0000","last_became_active":"2024-03-31T15:53:47.763310+0000","last_became_peered":"2024-03-31T15:53:47.763310+0000","last_unstale":"2024-03-31T15:53:52.921162+0000","last_undegraded":"2024-03-31T15:53:52.921162+0000","last_fullsized":"2024-03-31T15:53:52.921162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792492+0000","last_change":"2024-03-31T15:53:47.765680+0000","last_active":"2024-03-31T15:53:52.792492+0000","last_peered":"2024-03-31T15:53:52.792492+0000","last_clean":"2024-03-31T15:53:52.792492+0000","last_became_active":"2024-03-31T15:53:47.765516+0000","last_became_peered":"2024-03-31T15:53:47.765516+0000","last_unstale":"2024-03-31T15:53:52.792492+0000","last_undegraded":"2024-03-31T15:53:52.792492+0000","last_fullsized":"2024-03-31T15:53:52.792492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:44:09.409693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":102303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":102143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":856064,"data_stored":693093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:54:12.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:54:13.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 Processor -- start 2024-03-31T15:54:13.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 -- start start 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f423c06e3e0 0x7f423c06cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423c15fbd0 con 0x7f423c15c020 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f423c06d480 con 0x7f423c06e3e0 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f4241e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f423c15feb0 con 0x7f423c06d040 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f423c06e3e0 0x7f423c06cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f423c06e3e0 0x7f423c06cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51168/0 (socket says 172.21.15.90:51168) 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 -- 172.21.15.90:0/3570777508 learned_addr learned my addr 172.21.15.90:0/3570777508 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 -- 172.21.15.90:0/3570777508 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 msgr2=0x7f423c0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423b400640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c0cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 -- 172.21.15.90:0/3570777508 >> v1:172.21.15.102:6789/0 conn(0x7f423c15c020 legacy=0x7f423c0adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423a000640 1 -- 172.21.15.90:0/3570777508 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c160190 con 0x7f423c06e3e0 2024-03-31T15:54:13.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.162+0000 7f423b400640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f423a000640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c06e3e0 0x7f423c06cb00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4228009870 tx=0x7f422801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c611cba45c6d0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4239600640 1 -- 172.21.15.90:0/3570777508 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42280ab620 con 0x7f423c06e3e0 2024-03-31T15:54:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4239600640 1 -- 172.21.15.90:0/3570777508 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42280b0070 con 0x7f423c06e3e0 2024-03-31T15:54:13.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4239600640 1 -- 172.21.15.90:0/3570777508 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42280ab9e0 con 0x7f423c06e3e0 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 -- 172.21.15.90:0/3570777508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c06e3e0 msgr2=0x7f423c06cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c06e3e0 0x7f423c06cb00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4228009870 tx=0x7f422801c460 comp rx=0 tx=0).stop 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 -- 172.21.15.90:0/3570777508 shutdown_connections 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 --2- 172.21.15.90:0/3570777508 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c06e3e0 0x7f423c06cb00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 -- 172.21.15.90:0/3570777508 >> 172.21.15.90:0/3570777508 conn(0x7f423c0d13a0 msgr2=0x7f423c0d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 -- 172.21.15.90:0/3570777508 shutdown_connections 2024-03-31T15:54:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.166+0000 7f4241e00640 1 -- 172.21.15.90:0/3570777508 wait complete. 2024-03-31T15:54:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 Processor -- start 2024-03-31T15:54:13.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- start start 2024-03-31T15:54:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423c06e3e0 0x7f423c0b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f423c0ae650 con 0x7f423c15c020 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f423c160a30 con 0x7f423c06e3e0 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f423c0ae7d0 con 0x7f423c06d040 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51172/0 (socket says 172.21.15.90:51172) 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 -- 172.21.15.90:0/707451311 learned_addr learned my addr 172.21.15.90:0/707451311 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423a000640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423c06e3e0 0x7f423c0b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423aa00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 -- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 msgr2=0x7f423c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c14d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 -- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423c06e3e0 msgr2=0x7f423c0b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423c06e3e0 0x7f423c0b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 -- 172.21.15.90:0/707451311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c14d8c0 con 0x7f423c15c020 2024-03-31T15:54:13.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f423b400640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f423400cd50 tx=0x7f423400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423400f680 con 0x7f423c15c020 2024-03-31T15:54:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423400f840 con 0x7f423c15c020 2024-03-31T15:54:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423c0b2540 con 0x7f423c15c020 2024-03-31T15:54:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423400fc30 con 0x7f423c15c020 2024-03-31T15:54:13.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423c15cf90 con 0x7f423c15c020 2024-03-31T15:54:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.170+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423c0686d0 con 0x7f423c15c020 2024-03-31T15:54:13.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.174+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42340213f0 con 0x7f423c15c020 2024-03-31T15:54:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.174+0000 7f4233400640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 0x7f421c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.174+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f42340878a0 con 0x7f423c15c020 2024-03-31T15:54:13.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.174+0000 7f423aa00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 0x7f421c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.174+0000 7f423aa00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 0x7f421c06a840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f422c007fd0 tx=0x7f422c007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.182+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4234014030 con 0x7f423c15c020 2024-03-31T15:54:13.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.338+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f423c0732e0 con 0x7f421c068380 2024-03-31T15:54:13.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.346+0000 7f4233400640 1 -- 172.21.15.90:0/707451311 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332538 (secure 0 0 0) 0x7f423c0732e0 con 0x7f421c068380 2024-03-31T15:54:13.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:13.349 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:54:13.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 msgr2=0x7f421c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 0x7f421c06a840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f422c007fd0 tx=0x7f422c007e40 comp rx=0 tx=0).stop 2024-03-31T15:54:13.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 msgr2=0x7f423c15d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f423400cd50 tx=0x7f423400a650 comp rx=0 tx=0).stop 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 shutdown_connections 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f421c068380 0x7f421c06a840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f423c15c020 0x7f423c15d790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f423c06e3e0 0x7f423c0b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 --2- 172.21.15.90:0/707451311 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f423c06d040 0x7f423c14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 >> 172.21.15.90:0/707451311 conn(0x7f423c0d13a0 msgr2=0x7f423c0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 shutdown_connections 2024-03-31T15:54:13.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.354+0000 7f4241e00640 1 -- 172.21.15.90:0/707451311 wait complete. 2024-03-31T15:54:13.375 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-31T15:54:13.221383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219416,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755304,"statfs":{"total":773094113280,"available":772869431296,"internally_reserved":0,"allocated":3317760,"data_stored":2007160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003950"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359793+0000","last_change":"2024-03-31T15:53:46.757375+0000","last_active":"2024-03-31T15:53:53.359793+0000","last_peered":"2024-03-31T15:53:53.359793+0000","last_clean":"2024-03-31T15:53:53.359793+0000","last_became_active":"2024-03-31T15:53:46.757092+0000","last_became_peered":"2024-03-31T15:53:46.757092+0000","last_unstale":"2024-03-31T15:53:53.359793+0000","last_undegraded":"2024-03-31T15:53:53.359793+0000","last_fullsized":"2024-03-31T15:53:53.359793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248347+0000","last_change":"2024-03-31T15:53:47.766865+0000","last_active":"2024-03-31T15:53:53.248347+0000","last_peered":"2024-03-31T15:53:53.248347+0000","last_clean":"2024-03-31T15:53:53.248347+0000","last_became_active":"2024-03-31T15:53:47.766202+0000","last_became_peered":"2024-03-31T15:53:47.766202+0000","last_unstale":"2024-03-31T15:53:53.248347+0000","last_undegraded":"2024-03-31T15:53:53.248347+0000","last_fullsized":"2024-03-31T15:53:53.248347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:31.997737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247733+0000","last_change":"2024-03-31T15:53:51.804152+0000","last_active":"2024-03-31T15:53:53.247733+0000","last_peered":"2024-03-31T15:53:53.247733+0000","last_clean":"2024-03-31T15:53:53.247733+0000","last_became_active":"2024-03-31T15:53:51.802511+0000","last_became_peered":"2024-03-31T15:53:51.802511+0000","last_unstale":"2024-03-31T15:53:53.247733+0000","last_undegraded":"2024-03-31T15:53:53.247733+0000","last_fullsized":"2024-03-31T15:53:53.247733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:38.965494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248547+0000","last_change":"2024-03-31T15:53:46.763714+0000","last_active":"2024-03-31T15:53:53.248547+0000","last_peered":"2024-03-31T15:53:53.248547+0000","last_clean":"2024-03-31T15:53:53.248547+0000","last_became_active":"2024-03-31T15:53:46.763376+0000","last_became_peered":"2024-03-31T15:53:46.763376+0000","last_unstale":"2024-03-31T15:53:53.248547+0000","last_undegraded":"2024-03-31T15:53:53.248547+0000","last_fullsized":"2024-03-31T15:53:53.248547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359675+0000","last_change":"2024-03-31T15:53:47.771438+0000","last_active":"2024-03-31T15:53:53.359675+0000","last_peered":"2024-03-31T15:53:53.359675+0000","last_clean":"2024-03-31T15:53:53.359675+0000","last_became_active":"2024-03-31T15:53:47.771175+0000","last_became_peered":"2024-03-31T15:53:47.771175+0000","last_unstale":"2024-03-31T15:53:53.359675+0000","last_undegraded":"2024-03-31T15:53:53.359675+0000","last_fullsized":"2024-03-31T15:53:53.359675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792499+0000","last_change":"2024-03-31T15:53:51.797656+0000","last_active":"2024-03-31T15:53:52.792499+0000","last_peered":"2024-03-31T15:53:52.792499+0000","last_clean":"2024-03-31T15:53:52.792499+0000","last_became_active":"2024-03-31T15:53:51.797356+0000","last_became_peered":"2024-03-31T15:53:51.797356+0000","last_unstale":"2024-03-31T15:53:52.792499+0000","last_undegraded":"2024-03-31T15:53:52.792499+0000","last_fullsized":"2024-03-31T15:53:52.792499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:05.027492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792090+0000","last_change":"2024-03-31T15:53:46.764787+0000","last_active":"2024-03-31T15:53:52.792090+0000","last_peered":"2024-03-31T15:53:52.792090+0000","last_clean":"2024-03-31T15:53:52.792090+0000","last_became_active":"2024-03-31T15:53:46.764484+0000","last_became_peered":"2024-03-31T15:53:46.764484+0000","last_unstale":"2024-03-31T15:53:52.792090+0000","last_undegraded":"2024-03-31T15:53:52.792090+0000","last_fullsized":"2024-03-31T15:53:52.792090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:15.262251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789729+0000","last_change":"2024-03-31T15:53:47.773460+0000","last_active":"2024-03-31T15:53:52.789729+0000","last_peered":"2024-03-31T15:53:52.789729+0000","last_clean":"2024-03-31T15:53:52.789729+0000","last_became_active":"2024-03-31T15:53:47.773148+0000","last_became_peered":"2024-03-31T15:53:47.773148+0000","last_unstale":"2024-03-31T15:53:52.789729+0000","last_undegraded":"2024-03-31T15:53:52.789729+0000","last_fullsized":"2024-03-31T15:53:52.789729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.396848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793233+0000","last_change":"2024-03-31T15:53:51.799743+0000","last_active":"2024-03-31T15:53:52.793233+0000","last_peered":"2024-03-31T15:53:52.793233+0000","last_clean":"2024-03-31T15:53:52.793233+0000","last_became_active":"2024-03-31T15:53:51.799332+0000","last_became_peered":"2024-03-31T15:53:51.799332+0000","last_unstale":"2024-03-31T15:53:52.793233+0000","last_undegraded":"2024-03-31T15:53:52.793233+0000","last_fullsized":"2024-03-31T15:53:52.793233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:58.431101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791817+0000","last_change":"2024-03-31T15:53:46.765429+0000","last_active":"2024-03-31T15:53:52.791817+0000","last_peered":"2024-03-31T15:53:52.791817+0000","last_clean":"2024-03-31T15:53:52.791817+0000","last_became_active":"2024-03-31T15:53:46.765129+0000","last_became_peered":"2024-03-31T15:53:46.765129+0000","last_unstale":"2024-03-31T15:53:52.791817+0000","last_undegraded":"2024-03-31T15:53:52.791817+0000","last_fullsized":"2024-03-31T15:53:52.791817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:00:22.460897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790207+0000","last_change":"2024-03-31T15:53:47.770281+0000","last_active":"2024-03-31T15:53:52.790207+0000","last_peered":"2024-03-31T15:53:52.790207+0000","last_clean":"2024-03-31T15:53:52.790207+0000","last_became_active":"2024-03-31T15:53:47.769991+0000","last_became_peered":"2024-03-31T15:53:47.769991+0000","last_unstale":"2024-03-31T15:53:52.790207+0000","last_undegraded":"2024-03-31T15:53:52.790207+0000","last_fullsized":"2024-03-31T15:53:52.790207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:20.227687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792734+0000","last_change":"2024-03-31T15:53:51.797162+0000","last_active":"2024-03-31T15:53:52.792734+0000","last_peered":"2024-03-31T15:53:52.792734+0000","last_clean":"2024-03-31T15:53:52.792734+0000","last_became_active":"2024-03-31T15:53:51.796702+0000","last_became_peered":"2024-03-31T15:53:51.796702+0000","last_unstale":"2024-03-31T15:53:52.792734+0000","last_undegraded":"2024-03-31T15:53:52.792734+0000","last_fullsized":"2024-03-31T15:53:52.792734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:40.593467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306348+0000","last_change":"2024-03-31T15:53:47.762689+0000","last_active":"2024-03-31T15:53:53.306348+0000","last_peered":"2024-03-31T15:53:53.306348+0000","last_clean":"2024-03-31T15:53:53.306348+0000","last_became_active":"2024-03-31T15:53:47.762427+0000","last_became_peered":"2024-03-31T15:53:47.762427+0000","last_unstale":"2024-03-31T15:53:53.306348+0000","last_undegraded":"2024-03-31T15:53:53.306348+0000","last_fullsized":"2024-03-31T15:53:53.306348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791816+0000","last_change":"2024-03-31T15:53:46.757859+0000","last_active":"2024-03-31T15:53:52.791816+0000","last_peered":"2024-03-31T15:53:52.791816+0000","last_clean":"2024-03-31T15:53:52.791816+0000","last_became_active":"2024-03-31T15:53:46.757618+0000","last_became_peered":"2024-03-31T15:53:46.757618+0000","last_unstale":"2024-03-31T15:53:52.791816+0000","last_undegraded":"2024-03-31T15:53:52.791816+0000","last_fullsized":"2024-03-31T15:53:52.791816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:08:18.634688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793519+0000","last_change":"2024-03-31T15:53:51.797915+0000","last_active":"2024-03-31T15:53:52.793519+0000","last_peered":"2024-03-31T15:53:52.793519+0000","last_clean":"2024-03-31T15:53:52.793519+0000","last_became_active":"2024-03-31T15:53:51.797515+0000","last_became_peered":"2024-03-31T15:53:51.797515+0000","last_unstale":"2024-03-31T15:53:52.793519+0000","last_undegraded":"2024-03-31T15:53:52.793519+0000","last_fullsized":"2024-03-31T15:53:52.793519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247998+0000","last_change":"2024-03-31T15:53:47.765185+0000","last_active":"2024-03-31T15:53:53.247998+0000","last_peered":"2024-03-31T15:53:53.247998+0000","last_clean":"2024-03-31T15:53:53.247998+0000","last_became_active":"2024-03-31T15:53:47.764777+0000","last_became_peered":"2024-03-31T15:53:47.764777+0000","last_unstale":"2024-03-31T15:53:53.247998+0000","last_undegraded":"2024-03-31T15:53:53.247998+0000","last_fullsized":"2024-03-31T15:53:53.247998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:23.224004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920654+0000","last_change":"2024-03-31T15:53:46.762534+0000","last_active":"2024-03-31T15:53:52.920654+0000","last_peered":"2024-03-31T15:53:52.920654+0000","last_clean":"2024-03-31T15:53:52.920654+0000","last_became_active":"2024-03-31T15:53:46.762081+0000","last_became_peered":"2024-03-31T15:53:46.762081+0000","last_unstale":"2024-03-31T15:53:52.920654+0000","last_undegraded":"2024-03-31T15:53:52.920654+0000","last_fullsized":"2024-03-31T15:53:52.920654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793296+0000","last_change":"2024-03-31T15:53:51.802744+0000","last_active":"2024-03-31T15:53:52.793296+0000","last_peered":"2024-03-31T15:53:52.793296+0000","last_clean":"2024-03-31T15:53:52.793296+0000","last_became_active":"2024-03-31T15:53:51.802328+0000","last_became_peered":"2024-03-31T15:53:51.802328+0000","last_unstale":"2024-03-31T15:53:52.793296+0000","last_undegraded":"2024-03-31T15:53:52.793296+0000","last_fullsized":"2024-03-31T15:53:52.793296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:00.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247904+0000","last_change":"2024-03-31T15:53:47.751398+0000","last_active":"2024-03-31T15:53:53.247904+0000","last_peered":"2024-03-31T15:53:53.247904+0000","last_clean":"2024-03-31T15:53:53.247904+0000","last_became_active":"2024-03-31T15:53:47.750758+0000","last_became_peered":"2024-03-31T15:53:47.750758+0000","last_unstale":"2024-03-31T15:53:53.247904+0000","last_undegraded":"2024-03-31T15:53:53.247904+0000","last_fullsized":"2024-03-31T15:53:53.247904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:31.275051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920257+0000","last_change":"2024-03-31T15:53:46.755068+0000","last_active":"2024-03-31T15:53:52.920257+0000","last_peered":"2024-03-31T15:53:52.920257+0000","last_clean":"2024-03-31T15:53:52.920257+0000","last_became_active":"2024-03-31T15:53:46.754754+0000","last_became_peered":"2024-03-31T15:53:46.754754+0000","last_unstale":"2024-03-31T15:53:52.920257+0000","last_undegraded":"2024-03-31T15:53:52.920257+0000","last_fullsized":"2024-03-31T15:53:52.920257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921340+0000","last_change":"2024-03-31T15:53:51.803182+0000","last_active":"2024-03-31T15:53:52.921340+0000","last_peered":"2024-03-31T15:53:52.921340+0000","last_clean":"2024-03-31T15:53:52.921340+0000","last_became_active":"2024-03-31T15:53:51.802710+0000","last_became_peered":"2024-03-31T15:53:51.802710+0000","last_unstale":"2024-03-31T15:53:52.921340+0000","last_undegraded":"2024-03-31T15:53:52.921340+0000","last_fullsized":"2024-03-31T15:53:52.921340+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:40.165393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792965+0000","last_change":"2024-03-31T15:53:47.755358+0000","last_active":"2024-03-31T15:53:52.792965+0000","last_peered":"2024-03-31T15:53:52.792965+0000","last_clean":"2024-03-31T15:53:52.792965+0000","last_became_active":"2024-03-31T15:53:47.754487+0000","last_became_peered":"2024-03-31T15:53:47.754487+0000","last_unstale":"2024-03-31T15:53:52.792965+0000","last_undegraded":"2024-03-31T15:53:52.792965+0000","last_fullsized":"2024-03-31T15:53:52.792965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:53.949585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791156+0000","last_change":"2024-03-31T15:53:46.755464+0000","last_active":"2024-03-31T15:53:52.791156+0000","last_peered":"2024-03-31T15:53:52.791156+0000","last_clean":"2024-03-31T15:53:52.791156+0000","last_became_active":"2024-03-31T15:53:46.754626+0000","last_became_peered":"2024-03-31T15:53:46.754626+0000","last_unstale":"2024-03-31T15:53:52.791156+0000","last_undegraded":"2024-03-31T15:53:52.791156+0000","last_fullsized":"2024-03-31T15:53:52.791156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:18.769264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793394+0000","last_change":"2024-03-31T15:53:51.803474+0000","last_active":"2024-03-31T15:53:52.793394+0000","last_peered":"2024-03-31T15:53:52.793394+0000","last_clean":"2024-03-31T15:53:52.793394+0000","last_became_active":"2024-03-31T15:53:51.803014+0000","last_became_peered":"2024-03-31T15:53:51.803014+0000","last_unstale":"2024-03-31T15:53:52.793394+0000","last_undegraded":"2024-03-31T15:53:52.793394+0000","last_fullsized":"2024-03-31T15:53:52.793394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359741+0000","last_change":"2024-03-31T15:53:47.754929+0000","last_active":"2024-03-31T15:53:53.359741+0000","last_peered":"2024-03-31T15:53:53.359741+0000","last_clean":"2024-03-31T15:53:53.359741+0000","last_became_active":"2024-03-31T15:53:47.754625+0000","last_became_peered":"2024-03-31T15:53:47.754625+0000","last_unstale":"2024-03-31T15:53:53.359741+0000","last_undegraded":"2024-03-31T15:53:53.359741+0000","last_fullsized":"2024-03-31T15:53:53.359741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307339+0000","last_change":"2024-03-31T15:53:46.757784+0000","last_active":"2024-03-31T15:53:53.307339+0000","last_peered":"2024-03-31T15:53:53.307339+0000","last_clean":"2024-03-31T15:53:53.307339+0000","last_became_active":"2024-03-31T15:53:46.757342+0000","last_became_peered":"2024-03-31T15:53:46.757342+0000","last_unstale":"2024-03-31T15:53:53.307339+0000","last_undegraded":"2024-03-31T15:53:53.307339+0000","last_fullsized":"2024-03-31T15:53:53.307339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792725+0000","last_change":"2024-03-31T15:53:51.800310+0000","last_active":"2024-03-31T15:53:52.792725+0000","last_peered":"2024-03-31T15:53:52.792725+0000","last_clean":"2024-03-31T15:53:52.792725+0000","last_became_active":"2024-03-31T15:53:51.798223+0000","last_became_peered":"2024-03-31T15:53:51.798223+0000","last_unstale":"2024-03-31T15:53:52.792725+0000","last_undegraded":"2024-03-31T15:53:52.792725+0000","last_fullsized":"2024-03-31T15:53:52.792725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248692+0000","last_change":"2024-03-31T15:53:47.754983+0000","last_active":"2024-03-31T15:53:53.248692+0000","last_peered":"2024-03-31T15:53:53.248692+0000","last_clean":"2024-03-31T15:53:53.248692+0000","last_became_active":"2024-03-31T15:53:47.754734+0000","last_became_peered":"2024-03-31T15:53:47.754734+0000","last_unstale":"2024-03-31T15:53:53.248692+0000","last_undegraded":"2024-03-31T15:53:53.248692+0000","last_fullsized":"2024-03-31T15:53:53.248692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:31:24.931497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921179+0000","last_change":"2024-03-31T15:53:46.763360+0000","last_active":"2024-03-31T15:53:52.921179+0000","last_peered":"2024-03-31T15:53:52.921179+0000","last_clean":"2024-03-31T15:53:52.921179+0000","last_became_active":"2024-03-31T15:53:46.763005+0000","last_became_peered":"2024-03-31T15:53:46.763005+0000","last_unstale":"2024-03-31T15:53:52.921179+0000","last_undegraded":"2024-03-31T15:53:52.921179+0000","last_fullsized":"2024-03-31T15:53:52.921179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:30.435282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247663+0000","last_change":"2024-03-31T15:53:51.800117+0000","last_active":"2024-03-31T15:53:53.247663+0000","last_peered":"2024-03-31T15:53:53.247663+0000","last_clean":"2024-03-31T15:53:53.247663+0000","last_became_active":"2024-03-31T15:53:51.799217+0000","last_became_peered":"2024-03-31T15:53:51.799217+0000","last_unstale":"2024-03-31T15:53:53.247663+0000","last_undegraded":"2024-03-31T15:53:53.247663+0000","last_fullsized":"2024-03-31T15:53:53.247663+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:39.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920855+0000","last_change":"2024-03-31T15:53:47.754428+0000","last_active":"2024-03-31T15:53:52.920855+0000","last_peered":"2024-03-31T15:53:52.920855+0000","last_clean":"2024-03-31T15:53:52.920855+0000","last_became_active":"2024-03-31T15:53:47.753085+0000","last_became_peered":"2024-03-31T15:53:47.753085+0000","last_unstale":"2024-03-31T15:53:52.920855+0000","last_undegraded":"2024-03-31T15:53:52.920855+0000","last_fullsized":"2024-03-31T15:53:52.920855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792251+0000","last_change":"2024-03-31T15:53:46.763802+0000","last_active":"2024-03-31T15:53:52.792251+0000","last_peered":"2024-03-31T15:53:52.792251+0000","last_clean":"2024-03-31T15:53:52.792251+0000","last_became_active":"2024-03-31T15:53:46.762025+0000","last_became_peered":"2024-03-31T15:53:46.762025+0000","last_unstale":"2024-03-31T15:53:52.792251+0000","last_undegraded":"2024-03-31T15:53:52.792251+0000","last_fullsized":"2024-03-31T15:53:52.792251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:50.235479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359173+0000","last_change":"2024-03-31T15:53:51.796387+0000","last_active":"2024-03-31T15:53:53.359173+0000","last_peered":"2024-03-31T15:53:53.359173+0000","last_clean":"2024-03-31T15:53:53.359173+0000","last_became_active":"2024-03-31T15:53:51.795890+0000","last_became_peered":"2024-03-31T15:53:51.795890+0000","last_unstale":"2024-03-31T15:53:53.359173+0000","last_undegraded":"2024-03-31T15:53:53.359173+0000","last_fullsized":"2024-03-31T15:53:53.359173+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790297+0000","last_change":"2024-03-31T15:53:47.765672+0000","last_active":"2024-03-31T15:53:52.790297+0000","last_peered":"2024-03-31T15:53:52.790297+0000","last_clean":"2024-03-31T15:53:52.790297+0000","last_became_active":"2024-03-31T15:53:47.765481+0000","last_became_peered":"2024-03-31T15:53:47.765481+0000","last_unstale":"2024-03-31T15:53:52.790297+0000","last_undegraded":"2024-03-31T15:53:52.790297+0000","last_fullsized":"2024-03-31T15:53:52.790297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:14.332486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920943+0000","last_change":"2024-03-31T15:53:46.752123+0000","last_active":"2024-03-31T15:53:52.920943+0000","last_peered":"2024-03-31T15:53:52.920943+0000","last_clean":"2024-03-31T15:53:52.920943+0000","last_became_active":"2024-03-31T15:53:46.751752+0000","last_became_peered":"2024-03-31T15:53:46.751752+0000","last_unstale":"2024-03-31T15:53:52.920943+0000","last_undegraded":"2024-03-31T15:53:52.920943+0000","last_fullsized":"2024-03-31T15:53:52.920943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920358+0000","last_change":"2024-03-31T15:53:51.798289+0000","last_active":"2024-03-31T15:53:52.920358+0000","last_peered":"2024-03-31T15:53:52.920358+0000","last_clean":"2024-03-31T15:53:52.920358+0000","last_became_active":"2024-03-31T15:53:51.797843+0000","last_became_peered":"2024-03-31T15:53:51.797843+0000","last_unstale":"2024-03-31T15:53:52.920358+0000","last_undegraded":"2024-03-31T15:53:52.920358+0000","last_fullsized":"2024-03-31T15:53:52.920358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793832+0000","last_change":"2024-03-31T15:53:47.746865+0000","last_active":"2024-03-31T15:53:52.793832+0000","last_peered":"2024-03-31T15:53:52.793832+0000","last_clean":"2024-03-31T15:53:52.793832+0000","last_became_active":"2024-03-31T15:53:47.746098+0000","last_became_peered":"2024-03-31T15:53:47.746098+0000","last_unstale":"2024-03-31T15:53:52.793832+0000","last_undegraded":"2024-03-31T15:53:52.793832+0000","last_fullsized":"2024-03-31T15:53:52.793832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:31.443157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790702+0000","last_change":"2024-03-31T15:53:46.753263+0000","last_active":"2024-03-31T15:53:52.790702+0000","last_peered":"2024-03-31T15:53:52.790702+0000","last_clean":"2024-03-31T15:53:52.790702+0000","last_became_active":"2024-03-31T15:53:46.751947+0000","last_became_peered":"2024-03-31T15:53:46.751947+0000","last_unstale":"2024-03-31T15:53:52.790702+0000","last_undegraded":"2024-03-31T15:53:52.790702+0000","last_fullsized":"2024-03-31T15:53:52.790702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920882+0000","last_change":"2024-03-31T15:53:51.797078+0000","last_active":"2024-03-31T15:53:52.920882+0000","last_peered":"2024-03-31T15:53:52.920882+0000","last_clean":"2024-03-31T15:53:52.920882+0000","last_became_active":"2024-03-31T15:53:51.796451+0000","last_became_peered":"2024-03-31T15:53:51.796451+0000","last_unstale":"2024-03-31T15:53:52.920882+0000","last_undegraded":"2024-03-31T15:53:52.920882+0000","last_fullsized":"2024-03-31T15:53:52.920882+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920377+0000","last_change":"2024-03-31T15:53:47.752652+0000","last_active":"2024-03-31T15:53:52.920377+0000","last_peered":"2024-03-31T15:53:52.920377+0000","last_clean":"2024-03-31T15:53:52.920377+0000","last_became_active":"2024-03-31T15:53:47.752105+0000","last_became_peered":"2024-03-31T15:53:47.752105+0000","last_unstale":"2024-03-31T15:53:52.920377+0000","last_undegraded":"2024-03-31T15:53:52.920377+0000","last_fullsized":"2024-03-31T15:53:52.920377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528393+0000","last_change":"2024-03-31T15:53:46.753745+0000","last_active":"2024-03-31T15:53:53.528393+0000","last_peered":"2024-03-31T15:53:53.528393+0000","last_clean":"2024-03-31T15:53:53.528393+0000","last_became_active":"2024-03-31T15:53:46.753423+0000","last_became_peered":"2024-03-31T15:53:46.753423+0000","last_unstale":"2024-03-31T15:53:53.528393+0000","last_undegraded":"2024-03-31T15:53:53.528393+0000","last_fullsized":"2024-03-31T15:53:53.528393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921329+0000","last_change":"2024-03-31T15:53:51.802368+0000","last_active":"2024-03-31T15:53:52.921329+0000","last_peered":"2024-03-31T15:53:52.921329+0000","last_clean":"2024-03-31T15:53:52.921329+0000","last_became_active":"2024-03-31T15:53:51.801946+0000","last_became_peered":"2024-03-31T15:53:51.801946+0000","last_unstale":"2024-03-31T15:53:52.921329+0000","last_undegraded":"2024-03-31T15:53:52.921329+0000","last_fullsized":"2024-03-31T15:53:52.921329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.358775+0000","last_change":"2024-03-31T15:53:47.745539+0000","last_active":"2024-03-31T15:53:53.358775+0000","last_peered":"2024-03-31T15:53:53.358775+0000","last_clean":"2024-03-31T15:53:53.358775+0000","last_became_active":"2024-03-31T15:53:47.745265+0000","last_became_peered":"2024-03-31T15:53:47.745265+0000","last_unstale":"2024-03-31T15:53:53.358775+0000","last_undegraded":"2024-03-31T15:53:53.358775+0000","last_fullsized":"2024-03-31T15:53:53.358775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:12:22.550188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791245+0000","last_change":"2024-03-31T15:53:46.761890+0000","last_active":"2024-03-31T15:53:52.791245+0000","last_peered":"2024-03-31T15:53:52.791245+0000","last_clean":"2024-03-31T15:53:52.791245+0000","last_became_active":"2024-03-31T15:53:46.761417+0000","last_became_peered":"2024-03-31T15:53:46.761417+0000","last_unstale":"2024-03-31T15:53:52.791245+0000","last_undegraded":"2024-03-31T15:53:52.791245+0000","last_fullsized":"2024-03-31T15:53:52.791245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:25.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359685+0000","last_change":"2024-03-31T15:53:51.796610+0000","last_active":"2024-03-31T15:53:53.359685+0000","last_peered":"2024-03-31T15:53:53.359685+0000","last_clean":"2024-03-31T15:53:53.359685+0000","last_became_active":"2024-03-31T15:53:51.796112+0000","last_became_peered":"2024-03-31T15:53:51.796112+0000","last_unstale":"2024-03-31T15:53:53.359685+0000","last_undegraded":"2024-03-31T15:53:53.359685+0000","last_fullsized":"2024-03-31T15:53:53.359685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247912+0000","last_change":"2024-03-31T15:53:47.768681+0000","last_active":"2024-03-31T15:53:53.247912+0000","last_peered":"2024-03-31T15:53:53.247912+0000","last_clean":"2024-03-31T15:53:53.247912+0000","last_became_active":"2024-03-31T15:53:47.768416+0000","last_became_peered":"2024-03-31T15:53:47.768416+0000","last_unstale":"2024-03-31T15:53:53.247912+0000","last_undegraded":"2024-03-31T15:53:53.247912+0000","last_fullsized":"2024-03-31T15:53:53.247912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:02.383327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789855+0000","last_change":"2024-03-31T15:53:46.760553+0000","last_active":"2024-03-31T15:53:52.789855+0000","last_peered":"2024-03-31T15:53:52.789855+0000","last_clean":"2024-03-31T15:53:52.789855+0000","last_became_active":"2024-03-31T15:53:46.760258+0000","last_became_peered":"2024-03-31T15:53:46.760258+0000","last_unstale":"2024-03-31T15:53:52.789855+0000","last_undegraded":"2024-03-31T15:53:52.789855+0000","last_fullsized":"2024-03-31T15:53:52.789855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791305+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:53:52.791305+0000","last_peered":"2024-03-31T15:53:52.791305+0000","last_clean":"2024-03-31T15:53:52.791305+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:53:52.791305+0000","last_undegraded":"2024-03-31T15:53:52.791305+0000","last_fullsized":"2024-03-31T15:53:52.791305+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306820+0000","last_change":"2024-03-31T15:53:51.789901+0000","last_active":"2024-03-31T15:53:53.306820+0000","last_peered":"2024-03-31T15:53:53.306820+0000","last_clean":"2024-03-31T15:53:53.306820+0000","last_became_active":"2024-03-31T15:53:51.789529+0000","last_became_peered":"2024-03-31T15:53:51.789529+0000","last_unstale":"2024-03-31T15:53:53.306820+0000","last_undegraded":"2024-03-31T15:53:53.306820+0000","last_fullsized":"2024-03-31T15:53:53.306820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791956+0000","last_change":"2024-03-31T15:53:47.764710+0000","last_active":"2024-03-31T15:53:52.791956+0000","last_peered":"2024-03-31T15:53:52.791956+0000","last_clean":"2024-03-31T15:53:52.791956+0000","last_became_active":"2024-03-31T15:53:47.764408+0000","last_became_peered":"2024-03-31T15:53:47.764408+0000","last_unstale":"2024-03-31T15:53:52.791956+0000","last_undegraded":"2024-03-31T15:53:52.791956+0000","last_fullsized":"2024-03-31T15:53:52.791956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:14:45.512051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528719+0000","last_change":"2024-03-31T15:53:46.757343+0000","last_active":"2024-03-31T15:53:53.528719+0000","last_peered":"2024-03-31T15:53:53.528719+0000","last_clean":"2024-03-31T15:53:53.528719+0000","last_became_active":"2024-03-31T15:53:46.757028+0000","last_became_peered":"2024-03-31T15:53:46.757028+0000","last_unstale":"2024-03-31T15:53:53.528719+0000","last_undegraded":"2024-03-31T15:53:53.528719+0000","last_fullsized":"2024-03-31T15:53:53.528719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:19.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306461+0000","last_change":"2024-03-31T15:53:51.799696+0000","last_active":"2024-03-31T15:53:53.306461+0000","last_peered":"2024-03-31T15:53:53.306461+0000","last_clean":"2024-03-31T15:53:53.306461+0000","last_became_active":"2024-03-31T15:53:51.799041+0000","last_became_peered":"2024-03-31T15:53:51.799041+0000","last_unstale":"2024-03-31T15:53:53.306461+0000","last_undegraded":"2024-03-31T15:53:53.306461+0000","last_fullsized":"2024-03-31T15:53:53.306461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789786+0000","last_change":"2024-03-31T15:53:47.767189+0000","last_active":"2024-03-31T15:53:52.789786+0000","last_peered":"2024-03-31T15:53:52.789786+0000","last_clean":"2024-03-31T15:53:52.789786+0000","last_became_active":"2024-03-31T15:53:47.766941+0000","last_became_peered":"2024-03-31T15:53:47.766941+0000","last_unstale":"2024-03-31T15:53:52.789786+0000","last_undegraded":"2024-03-31T15:53:52.789786+0000","last_fullsized":"2024-03-31T15:53:52.789786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:56.064763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921010+0000","last_change":"2024-03-31T15:53:46.755784+0000","last_active":"2024-03-31T15:53:52.921010+0000","last_peered":"2024-03-31T15:53:52.921010+0000","last_clean":"2024-03-31T15:53:52.921010+0000","last_became_active":"2024-03-31T15:53:46.755518+0000","last_became_peered":"2024-03-31T15:53:46.755518+0000","last_unstale":"2024-03-31T15:53:52.921010+0000","last_undegraded":"2024-03-31T15:53:52.921010+0000","last_fullsized":"2024-03-31T15:53:52.921010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:03.068278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359266+0000","last_change":"2024-03-31T15:53:51.805030+0000","last_active":"2024-03-31T15:53:53.359266+0000","last_peered":"2024-03-31T15:53:53.359266+0000","last_clean":"2024-03-31T15:53:53.359266+0000","last_became_active":"2024-03-31T15:53:51.804608+0000","last_became_peered":"2024-03-31T15:53:51.804608+0000","last_unstale":"2024-03-31T15:53:53.359266+0000","last_undegraded":"2024-03-31T15:53:53.359266+0000","last_fullsized":"2024-03-31T15:53:53.359266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792768+0000","last_change":"2024-03-31T15:53:47.763067+0000","last_active":"2024-03-31T15:53:52.792768+0000","last_peered":"2024-03-31T15:53:52.792768+0000","last_clean":"2024-03-31T15:53:52.792768+0000","last_became_active":"2024-03-31T15:53:47.762801+0000","last_became_peered":"2024-03-31T15:53:47.762801+0000","last_unstale":"2024-03-31T15:53:52.792768+0000","last_undegraded":"2024-03-31T15:53:52.792768+0000","last_fullsized":"2024-03-31T15:53:52.792768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:34.931240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528883+0000","last_change":"2024-03-31T15:53:46.763307+0000","last_active":"2024-03-31T15:53:53.528883+0000","last_peered":"2024-03-31T15:53:53.528883+0000","last_clean":"2024-03-31T15:53:53.528883+0000","last_became_active":"2024-03-31T15:53:46.763083+0000","last_became_peered":"2024-03-31T15:53:46.763083+0000","last_unstale":"2024-03-31T15:53:53.528883+0000","last_undegraded":"2024-03-31T15:53:53.528883+0000","last_fullsized":"2024-03-31T15:53:53.528883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:35:35.431822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789351+0000","last_change":"2024-03-31T15:53:51.792009+0000","last_active":"2024-03-31T15:53:52.789351+0000","last_peered":"2024-03-31T15:53:52.789351+0000","last_clean":"2024-03-31T15:53:52.789351+0000","last_became_active":"2024-03-31T15:53:51.791660+0000","last_became_peered":"2024-03-31T15:53:51.791660+0000","last_unstale":"2024-03-31T15:53:52.789351+0000","last_undegraded":"2024-03-31T15:53:52.789351+0000","last_fullsized":"2024-03-31T15:53:52.789351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:17.783876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792256+0000","last_change":"2024-03-31T15:53:47.771025+0000","last_active":"2024-03-31T15:53:52.792256+0000","last_peered":"2024-03-31T15:53:52.792256+0000","last_clean":"2024-03-31T15:53:52.792256+0000","last_became_active":"2024-03-31T15:53:47.769254+0000","last_became_peered":"2024-03-31T15:53:47.769254+0000","last_unstale":"2024-03-31T15:53:52.792256+0000","last_undegraded":"2024-03-31T15:53:52.792256+0000","last_fullsized":"2024-03-31T15:53:52.792256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:01.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793608+0000","last_change":"2024-03-31T15:53:46.759316+0000","last_active":"2024-03-31T15:53:52.793608+0000","last_peered":"2024-03-31T15:53:52.793608+0000","last_clean":"2024-03-31T15:53:52.793608+0000","last_became_active":"2024-03-31T15:53:46.758973+0000","last_became_peered":"2024-03-31T15:53:46.758973+0000","last_unstale":"2024-03-31T15:53:52.793608+0000","last_undegraded":"2024-03-31T15:53:52.793608+0000","last_fullsized":"2024-03-31T15:53:52.793608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:39.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792641+0000","last_change":"2024-03-31T15:53:51.795201+0000","last_active":"2024-03-31T15:53:52.792641+0000","last_peered":"2024-03-31T15:53:52.792641+0000","last_clean":"2024-03-31T15:53:52.792641+0000","last_became_active":"2024-03-31T15:53:51.794775+0000","last_became_peered":"2024-03-31T15:53:51.794775+0000","last_unstale":"2024-03-31T15:53:52.792641+0000","last_undegraded":"2024-03-31T15:53:52.792641+0000","last_fullsized":"2024-03-31T15:53:52.792641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:11.574086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791547+0000","last_change":"2024-03-31T15:53:47.760294+0000","last_active":"2024-03-31T15:53:52.791547+0000","last_peered":"2024-03-31T15:53:52.791547+0000","last_clean":"2024-03-31T15:53:52.791547+0000","last_became_active":"2024-03-31T15:53:47.760011+0000","last_became_peered":"2024-03-31T15:53:47.760011+0000","last_unstale":"2024-03-31T15:53:52.791547+0000","last_undegraded":"2024-03-31T15:53:52.791547+0000","last_fullsized":"2024-03-31T15:53:52.791547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:11.819602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528298+0000","last_change":"2024-03-31T15:53:46.759221+0000","last_active":"2024-03-31T15:53:53.528298+0000","last_peered":"2024-03-31T15:53:53.528298+0000","last_clean":"2024-03-31T15:53:53.528298+0000","last_became_active":"2024-03-31T15:53:46.758926+0000","last_became_peered":"2024-03-31T15:53:46.758926+0000","last_unstale":"2024-03-31T15:53:53.528298+0000","last_undegraded":"2024-03-31T15:53:53.528298+0000","last_fullsized":"2024-03-31T15:53:53.528298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359571+0000","last_change":"2024-03-31T15:53:51.794069+0000","last_active":"2024-03-31T15:53:53.359571+0000","last_peered":"2024-03-31T15:53:53.359571+0000","last_clean":"2024-03-31T15:53:53.359571+0000","last_became_active":"2024-03-31T15:53:51.793665+0000","last_became_peered":"2024-03-31T15:53:51.793665+0000","last_unstale":"2024-03-31T15:53:53.359571+0000","last_undegraded":"2024-03-31T15:53:53.359571+0000","last_fullsized":"2024-03-31T15:53:53.359571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306482+0000","last_change":"2024-03-31T15:53:47.763992+0000","last_active":"2024-03-31T15:53:53.306482+0000","last_peered":"2024-03-31T15:53:53.306482+0000","last_clean":"2024-03-31T15:53:53.306482+0000","last_became_active":"2024-03-31T15:53:47.763743+0000","last_became_peered":"2024-03-31T15:53:47.763743+0000","last_unstale":"2024-03-31T15:53:53.306482+0000","last_undegraded":"2024-03-31T15:53:53.306482+0000","last_fullsized":"2024-03-31T15:53:53.306482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306483+0000","last_change":"2024-03-31T15:53:46.764342+0000","last_active":"2024-03-31T15:53:53.306483+0000","last_peered":"2024-03-31T15:53:53.306483+0000","last_clean":"2024-03-31T15:53:53.306483+0000","last_became_active":"2024-03-31T15:53:46.763064+0000","last_became_peered":"2024-03-31T15:53:46.763064+0000","last_unstale":"2024-03-31T15:53:53.306483+0000","last_undegraded":"2024-03-31T15:53:53.306483+0000","last_fullsized":"2024-03-31T15:53:53.306483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248662+0000","last_change":"2024-03-31T15:53:51.802982+0000","last_active":"2024-03-31T15:53:53.248662+0000","last_peered":"2024-03-31T15:53:53.248662+0000","last_clean":"2024-03-31T15:53:53.248662+0000","last_became_active":"2024-03-31T15:53:51.802619+0000","last_became_peered":"2024-03-31T15:53:51.802619+0000","last_unstale":"2024-03-31T15:53:53.248662+0000","last_undegraded":"2024-03-31T15:53:53.248662+0000","last_fullsized":"2024-03-31T15:53:53.248662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:12:18.187743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791793+0000","last_change":"2024-03-31T15:53:47.763046+0000","last_active":"2024-03-31T15:53:52.791793+0000","last_peered":"2024-03-31T15:53:52.791793+0000","last_clean":"2024-03-31T15:53:52.791793+0000","last_became_active":"2024-03-31T15:53:47.762791+0000","last_became_peered":"2024-03-31T15:53:47.762791+0000","last_unstale":"2024-03-31T15:53:52.791793+0000","last_undegraded":"2024-03-31T15:53:52.791793+0000","last_fullsized":"2024-03-31T15:53:52.791793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:32:01.910382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790646+0000","last_change":"2024-03-31T15:53:46.765124+0000","last_active":"2024-03-31T15:53:52.790646+0000","last_peered":"2024-03-31T15:53:52.790646+0000","last_clean":"2024-03-31T15:53:52.790646+0000","last_became_active":"2024-03-31T15:53:46.764856+0000","last_became_peered":"2024-03-31T15:53:46.764856+0000","last_unstale":"2024-03-31T15:53:52.790646+0000","last_undegraded":"2024-03-31T15:53:52.790646+0000","last_fullsized":"2024-03-31T15:53:52.790646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248292+0000","last_change":"2024-03-31T15:53:51.807271+0000","last_active":"2024-03-31T15:53:53.248292+0000","last_peered":"2024-03-31T15:53:53.248292+0000","last_clean":"2024-03-31T15:53:53.248292+0000","last_became_active":"2024-03-31T15:53:51.806909+0000","last_became_peered":"2024-03-31T15:53:51.806909+0000","last_unstale":"2024-03-31T15:53:53.248292+0000","last_undegraded":"2024-03-31T15:53:53.248292+0000","last_fullsized":"2024-03-31T15:53:53.248292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:14:12.015472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359081+0000","last_change":"2024-03-31T15:53:47.762410+0000","last_active":"2024-03-31T15:53:53.359081+0000","last_peered":"2024-03-31T15:53:53.359081+0000","last_clean":"2024-03-31T15:53:53.359081+0000","last_became_active":"2024-03-31T15:53:47.762145+0000","last_became_peered":"2024-03-31T15:53:47.762145+0000","last_unstale":"2024-03-31T15:53:53.359081+0000","last_undegraded":"2024-03-31T15:53:53.359081+0000","last_fullsized":"2024-03-31T15:53:53.359081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359350+0000","last_change":"2024-03-31T15:53:46.756308+0000","last_active":"2024-03-31T15:53:53.359350+0000","last_peered":"2024-03-31T15:53:53.359350+0000","last_clean":"2024-03-31T15:53:53.359350+0000","last_became_active":"2024-03-31T15:53:46.756010+0000","last_became_peered":"2024-03-31T15:53:46.756010+0000","last_unstale":"2024-03-31T15:53:53.359350+0000","last_undegraded":"2024-03-31T15:53:53.359350+0000","last_fullsized":"2024-03-31T15:53:53.359350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:30:06.680148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790084+0000","last_change":"2024-03-31T15:53:51.806516+0000","last_active":"2024-03-31T15:53:52.790084+0000","last_peered":"2024-03-31T15:53:52.790084+0000","last_clean":"2024-03-31T15:53:52.790084+0000","last_became_active":"2024-03-31T15:53:51.805985+0000","last_became_peered":"2024-03-31T15:53:51.805985+0000","last_unstale":"2024-03-31T15:53:52.790084+0000","last_undegraded":"2024-03-31T15:53:52.790084+0000","last_fullsized":"2024-03-31T15:53:52.790084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:12:27.917062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791963+0000","last_change":"2024-03-31T15:53:47.768000+0000","last_active":"2024-03-31T15:53:52.791963+0000","last_peered":"2024-03-31T15:53:52.791963+0000","last_clean":"2024-03-31T15:53:52.791963+0000","last_became_active":"2024-03-31T15:53:47.767710+0000","last_became_peered":"2024-03-31T15:53:47.767710+0000","last_unstale":"2024-03-31T15:53:52.791963+0000","last_undegraded":"2024-03-31T15:53:52.791963+0000","last_fullsized":"2024-03-31T15:53:52.791963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:35.607484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:54:02.857882+0000","last_change":"2024-03-31T15:53:46.764364+0000","last_active":"2024-03-31T15:54:02.857882+0000","last_peered":"2024-03-31T15:54:02.857882+0000","last_clean":"2024-03-31T15:54:02.857882+0000","last_became_active":"2024-03-31T15:53:46.763234+0000","last_became_peered":"2024-03-31T15:53:46.763234+0000","last_unstale":"2024-03-31T15:54:02.857882+0000","last_undegraded":"2024-03-31T15:54:02.857882+0000","last_fullsized":"2024-03-31T15:54:02.857882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920563+0000","last_change":"2024-03-31T15:53:51.794393+0000","last_active":"2024-03-31T15:53:52.920563+0000","last_peered":"2024-03-31T15:53:52.920563+0000","last_clean":"2024-03-31T15:53:52.920563+0000","last_became_active":"2024-03-31T15:53:51.793984+0000","last_became_peered":"2024-03-31T15:53:51.793984+0000","last_unstale":"2024-03-31T15:53:52.920563+0000","last_undegraded":"2024-03-31T15:53:52.920563+0000","last_fullsized":"2024-03-31T15:53:52.920563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793726+0000","last_change":"2024-03-31T15:53:47.759150+0000","last_active":"2024-03-31T15:53:52.793726+0000","last_peered":"2024-03-31T15:53:52.793726+0000","last_clean":"2024-03-31T15:53:52.793726+0000","last_became_active":"2024-03-31T15:53:47.757846+0000","last_became_peered":"2024-03-31T15:53:47.757846+0000","last_unstale":"2024-03-31T15:53:52.793726+0000","last_undegraded":"2024-03-31T15:53:52.793726+0000","last_fullsized":"2024-03-31T15:53:52.793726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:16.643130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920932+0000","last_change":"2024-03-31T15:53:46.763645+0000","last_active":"2024-03-31T15:53:52.920932+0000","last_peered":"2024-03-31T15:53:52.920932+0000","last_clean":"2024-03-31T15:53:52.920932+0000","last_became_active":"2024-03-31T15:53:46.763370+0000","last_became_peered":"2024-03-31T15:53:46.763370+0000","last_unstale":"2024-03-31T15:53:52.920932+0000","last_undegraded":"2024-03-31T15:53:52.920932+0000","last_fullsized":"2024-03-31T15:53:52.920932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:15.635255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793545+0000","last_change":"2024-03-31T15:53:51.804406+0000","last_active":"2024-03-31T15:53:52.793545+0000","last_peered":"2024-03-31T15:53:52.793545+0000","last_clean":"2024-03-31T15:53:52.793545+0000","last_became_active":"2024-03-31T15:53:51.803887+0000","last_became_peered":"2024-03-31T15:53:51.803887+0000","last_unstale":"2024-03-31T15:53:52.793545+0000","last_undegraded":"2024-03-31T15:53:52.793545+0000","last_fullsized":"2024-03-31T15:53:52.793545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359778+0000","last_change":"2024-03-31T15:53:47.771467+0000","last_active":"2024-03-31T15:53:53.359778+0000","last_peered":"2024-03-31T15:53:53.359778+0000","last_clean":"2024-03-31T15:53:53.359778+0000","last_became_active":"2024-03-31T15:53:47.771267+0000","last_became_peered":"2024-03-31T15:53:47.771267+0000","last_unstale":"2024-03-31T15:53:53.359778+0000","last_undegraded":"2024-03-31T15:53:53.359778+0000","last_fullsized":"2024-03-31T15:53:53.359778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790236+0000","last_change":"2024-03-31T15:53:46.758836+0000","last_active":"2024-03-31T15:53:52.790236+0000","last_peered":"2024-03-31T15:53:52.790236+0000","last_clean":"2024-03-31T15:53:52.790236+0000","last_became_active":"2024-03-31T15:53:46.758359+0000","last_became_peered":"2024-03-31T15:53:46.758359+0000","last_unstale":"2024-03-31T15:53:52.790236+0000","last_undegraded":"2024-03-31T15:53:52.790236+0000","last_fullsized":"2024-03-31T15:53:52.790236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247843+0000","last_change":"2024-03-31T15:53:51.800437+0000","last_active":"2024-03-31T15:53:53.247843+0000","last_peered":"2024-03-31T15:53:53.247843+0000","last_clean":"2024-03-31T15:53:53.247843+0000","last_became_active":"2024-03-31T15:53:51.799728+0000","last_became_peered":"2024-03-31T15:53:51.799728+0000","last_unstale":"2024-03-31T15:53:53.247843+0000","last_undegraded":"2024-03-31T15:53:53.247843+0000","last_fullsized":"2024-03-31T15:53:53.247843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:45.272362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790191+0000","last_change":"2024-03-31T15:53:47.765037+0000","last_active":"2024-03-31T15:53:52.790191+0000","last_peered":"2024-03-31T15:53:52.790191+0000","last_clean":"2024-03-31T15:53:52.790191+0000","last_became_active":"2024-03-31T15:53:47.764764+0000","last_became_peered":"2024-03-31T15:53:47.764764+0000","last_unstale":"2024-03-31T15:53:52.790191+0000","last_undegraded":"2024-03-31T15:53:52.790191+0000","last_fullsized":"2024-03-31T15:53:52.790191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:07.287524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307175+0000","last_change":"2024-03-31T15:53:46.765922+0000","last_active":"2024-03-31T15:53:53.307175+0000","last_peered":"2024-03-31T15:53:53.307175+0000","last_clean":"2024-03-31T15:53:53.307175+0000","last_became_active":"2024-03-31T15:53:46.765530+0000","last_became_peered":"2024-03-31T15:53:46.765530+0000","last_unstale":"2024-03-31T15:53:53.307175+0000","last_undegraded":"2024-03-31T15:53:53.307175+0000","last_fullsized":"2024-03-31T15:53:53.307175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247784+0000","last_change":"2024-03-31T15:53:51.803591+0000","last_active":"2024-03-31T15:53:53.247784+0000","last_peered":"2024-03-31T15:53:53.247784+0000","last_clean":"2024-03-31T15:53:53.247784+0000","last_became_active":"2024-03-31T15:53:51.803139+0000","last_became_peered":"2024-03-31T15:53:51.803139+0000","last_unstale":"2024-03-31T15:53:53.247784+0000","last_undegraded":"2024-03-31T15:53:53.247784+0000","last_fullsized":"2024-03-31T15:53:53.247784+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:08.680601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359558+0000","last_change":"2024-03-31T15:53:51.809399+0000","last_active":"2024-03-31T15:53:53.359558+0000","last_peered":"2024-03-31T15:53:53.359558+0000","last_clean":"2024-03-31T15:53:53.359558+0000","last_became_active":"2024-03-31T15:53:51.808396+0000","last_became_peered":"2024-03-31T15:53:51.808396+0000","last_unstale":"2024-03-31T15:53:53.359558+0000","last_undegraded":"2024-03-31T15:53:53.359558+0000","last_fullsized":"2024-03-31T15:53:53.359558+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920147+0000","last_change":"2024-03-31T15:53:47.770618+0000","last_active":"2024-03-31T15:53:52.920147+0000","last_peered":"2024-03-31T15:53:52.920147+0000","last_clean":"2024-03-31T15:53:52.920147+0000","last_became_active":"2024-03-31T15:53:47.770333+0000","last_became_peered":"2024-03-31T15:53:47.770333+0000","last_unstale":"2024-03-31T15:53:52.920147+0000","last_undegraded":"2024-03-31T15:53:52.920147+0000","last_fullsized":"2024-03-31T15:53:52.920147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789491+0000","last_change":"2024-03-31T15:53:46.759181+0000","last_active":"2024-03-31T15:53:52.789491+0000","last_peered":"2024-03-31T15:53:52.789491+0000","last_clean":"2024-03-31T15:53:52.789491+0000","last_became_active":"2024-03-31T15:53:46.758355+0000","last_became_peered":"2024-03-31T15:53:46.758355+0000","last_unstale":"2024-03-31T15:53:52.789491+0000","last_undegraded":"2024-03-31T15:53:52.789491+0000","last_fullsized":"2024-03-31T15:53:52.789491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306986+0000","last_change":"2024-03-31T15:53:51.796763+0000","last_active":"2024-03-31T15:53:53.306986+0000","last_peered":"2024-03-31T15:53:53.306986+0000","last_clean":"2024-03-31T15:53:53.306986+0000","last_became_active":"2024-03-31T15:53:51.796351+0000","last_became_peered":"2024-03-31T15:53:51.796351+0000","last_unstale":"2024-03-31T15:53:53.306986+0000","last_undegraded":"2024-03-31T15:53:53.306986+0000","last_fullsized":"2024-03-31T15:53:53.306986+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247797+0000","last_change":"2024-03-31T15:53:47.766443+0000","last_active":"2024-03-31T15:53:53.247797+0000","last_peered":"2024-03-31T15:53:53.247797+0000","last_clean":"2024-03-31T15:53:53.247797+0000","last_became_active":"2024-03-31T15:53:47.766107+0000","last_became_peered":"2024-03-31T15:53:47.766107+0000","last_unstale":"2024-03-31T15:53:53.247797+0000","last_undegraded":"2024-03-31T15:53:53.247797+0000","last_fullsized":"2024-03-31T15:53:53.247797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:50:47.205172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248000+0000","last_change":"2024-03-31T15:53:46.764698+0000","last_active":"2024-03-31T15:53:53.248000+0000","last_peered":"2024-03-31T15:53:53.248000+0000","last_clean":"2024-03-31T15:53:53.248000+0000","last_became_active":"2024-03-31T15:53:46.764345+0000","last_became_peered":"2024-03-31T15:53:46.764345+0000","last_unstale":"2024-03-31T15:53:53.248000+0000","last_undegraded":"2024-03-31T15:53:53.248000+0000","last_fullsized":"2024-03-31T15:53:53.248000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:42.990113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306648+0000","last_change":"2024-03-31T15:53:46.765106+0000","last_active":"2024-03-31T15:53:53.306648+0000","last_peered":"2024-03-31T15:53:53.306648+0000","last_clean":"2024-03-31T15:53:53.306648+0000","last_became_active":"2024-03-31T15:53:46.764281+0000","last_became_peered":"2024-03-31T15:53:46.764281+0000","last_unstale":"2024-03-31T15:53:53.306648+0000","last_undegraded":"2024-03-31T15:53:53.306648+0000","last_fullsized":"2024-03-31T15:53:53.306648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359310+0000","last_change":"2024-03-31T15:53:47.757501+0000","last_active":"2024-03-31T15:53:53.359310+0000","last_peered":"2024-03-31T15:53:53.359310+0000","last_clean":"2024-03-31T15:53:53.359310+0000","last_became_active":"2024-03-31T15:53:47.757160+0000","last_became_peered":"2024-03-31T15:53:47.757160+0000","last_unstale":"2024-03-31T15:53:53.359310+0000","last_undegraded":"2024-03-31T15:53:53.359310+0000","last_fullsized":"2024-03-31T15:53:53.359310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528662+0000","last_change":"2024-03-31T15:53:51.794610+0000","last_active":"2024-03-31T15:53:53.528662+0000","last_peered":"2024-03-31T15:53:53.528662+0000","last_clean":"2024-03-31T15:53:53.528662+0000","last_became_active":"2024-03-31T15:53:51.794228+0000","last_became_peered":"2024-03-31T15:53:51.794228+0000","last_unstale":"2024-03-31T15:53:53.528662+0000","last_undegraded":"2024-03-31T15:53:53.528662+0000","last_fullsized":"2024-03-31T15:53:53.528662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:55.277607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248028+0000","last_change":"2024-03-31T15:53:46.762539+0000","last_active":"2024-03-31T15:53:53.248028+0000","last_peered":"2024-03-31T15:53:53.248028+0000","last_clean":"2024-03-31T15:53:53.248028+0000","last_became_active":"2024-03-31T15:53:46.761475+0000","last_became_peered":"2024-03-31T15:53:46.761475+0000","last_unstale":"2024-03-31T15:53:53.248028+0000","last_undegraded":"2024-03-31T15:53:53.248028+0000","last_fullsized":"2024-03-31T15:53:53.248028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:36:29.320997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793114+0000","last_change":"2024-03-31T15:53:47.758390+0000","last_active":"2024-03-31T15:53:52.793114+0000","last_peered":"2024-03-31T15:53:52.793114+0000","last_clean":"2024-03-31T15:53:52.793114+0000","last_became_active":"2024-03-31T15:53:47.754114+0000","last_became_peered":"2024-03-31T15:53:47.754114+0000","last_unstale":"2024-03-31T15:53:52.793114+0000","last_undegraded":"2024-03-31T15:53:52.793114+0000","last_fullsized":"2024-03-31T15:53:52.793114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:12.842845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248390+0000","last_change":"2024-03-31T15:53:51.799618+0000","last_active":"2024-03-31T15:53:53.248390+0000","last_peered":"2024-03-31T15:53:53.248390+0000","last_clean":"2024-03-31T15:53:53.248390+0000","last_became_active":"2024-03-31T15:53:51.799171+0000","last_became_peered":"2024-03-31T15:53:51.799171+0000","last_unstale":"2024-03-31T15:53:53.248390+0000","last_undegraded":"2024-03-31T15:53:53.248390+0000","last_fullsized":"2024-03-31T15:53:53.248390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:48.155512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306339+0000","last_change":"2024-03-31T15:53:47.765092+0000","last_active":"2024-03-31T15:53:53.306339+0000","last_peered":"2024-03-31T15:53:53.306339+0000","last_clean":"2024-03-31T15:53:53.306339+0000","last_became_active":"2024-03-31T15:53:47.764701+0000","last_became_peered":"2024-03-31T15:53:47.764701+0000","last_unstale":"2024-03-31T15:53:53.306339+0000","last_undegraded":"2024-03-31T15:53:53.306339+0000","last_fullsized":"2024-03-31T15:53:53.306339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528367+0000","last_change":"2024-03-31T15:53:47.765373+0000","last_active":"2024-03-31T15:53:53.528367+0000","last_peered":"2024-03-31T15:53:53.528367+0000","last_clean":"2024-03-31T15:53:53.528367+0000","last_became_active":"2024-03-31T15:53:47.764321+0000","last_became_peered":"2024-03-31T15:53:47.764321+0000","last_unstale":"2024-03-31T15:53:53.528367+0000","last_undegraded":"2024-03-31T15:53:53.528367+0000","last_fullsized":"2024-03-31T15:53:53.528367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:19.777139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790598+0000","last_change":"2024-03-31T15:53:47.762666+0000","last_active":"2024-03-31T15:53:52.790598+0000","last_peered":"2024-03-31T15:53:52.790598+0000","last_clean":"2024-03-31T15:53:52.790598+0000","last_became_active":"2024-03-31T15:53:47.762284+0000","last_became_peered":"2024-03-31T15:53:47.762284+0000","last_unstale":"2024-03-31T15:53:52.790598+0000","last_undegraded":"2024-03-31T15:53:52.790598+0000","last_fullsized":"2024-03-31T15:53:52.790598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:44.058276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359256+0000","last_change":"2024-03-31T15:53:47.762200+0000","last_active":"2024-03-31T15:53:53.359256+0000","last_peered":"2024-03-31T15:53:53.359256+0000","last_clean":"2024-03-31T15:53:53.359256+0000","last_became_active":"2024-03-31T15:53:47.761993+0000","last_became_peered":"2024-03-31T15:53:47.761993+0000","last_unstale":"2024-03-31T15:53:53.359256+0000","last_undegraded":"2024-03-31T15:53:53.359256+0000","last_fullsized":"2024-03-31T15:53:53.359256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248436+0000","last_change":"2024-03-31T15:53:47.758308+0000","last_active":"2024-03-31T15:53:53.248436+0000","last_peered":"2024-03-31T15:53:53.248436+0000","last_clean":"2024-03-31T15:53:53.248436+0000","last_became_active":"2024-03-31T15:53:47.757996+0000","last_became_peered":"2024-03-31T15:53:47.757996+0000","last_unstale":"2024-03-31T15:53:53.248436+0000","last_undegraded":"2024-03-31T15:53:53.248436+0000","last_fullsized":"2024-03-31T15:53:53.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:24:07.611521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920834+0000","last_change":"2024-03-31T15:53:47.754477+0000","last_active":"2024-03-31T15:53:52.920834+0000","last_peered":"2024-03-31T15:53:52.920834+0000","last_clean":"2024-03-31T15:53:52.920834+0000","last_became_active":"2024-03-31T15:53:47.754071+0000","last_became_peered":"2024-03-31T15:53:47.754071+0000","last_unstale":"2024-03-31T15:53:52.920834+0000","last_undegraded":"2024-03-31T15:53:52.920834+0000","last_fullsized":"2024-03-31T15:53:52.920834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307045+0000","last_change":"2024-03-31T15:53:47.761169+0000","last_active":"2024-03-31T15:53:53.307045+0000","last_peered":"2024-03-31T15:53:53.307045+0000","last_clean":"2024-03-31T15:53:53.307045+0000","last_became_active":"2024-03-31T15:53:47.760902+0000","last_became_peered":"2024-03-31T15:53:47.760902+0000","last_unstale":"2024-03-31T15:53:53.307045+0000","last_undegraded":"2024-03-31T15:53:53.307045+0000","last_fullsized":"2024-03-31T15:53:53.307045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248647+0000","last_change":"2024-03-31T15:53:47.758495+0000","last_active":"2024-03-31T15:53:53.248647+0000","last_peered":"2024-03-31T15:53:53.248647+0000","last_clean":"2024-03-31T15:53:53.248647+0000","last_became_active":"2024-03-31T15:53:47.758146+0000","last_became_peered":"2024-03-31T15:53:47.758146+0000","last_unstale":"2024-03-31T15:53:53.248647+0000","last_undegraded":"2024-03-31T15:53:53.248647+0000","last_fullsized":"2024-03-31T15:53:53.248647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:40:45.579157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789642+0000","last_change":"2024-03-31T15:53:47.759644+0000","last_active":"2024-03-31T15:53:52.789642+0000","last_peered":"2024-03-31T15:53:52.789642+0000","last_clean":"2024-03-31T15:53:52.789642+0000","last_became_active":"2024-03-31T15:53:47.759278+0000","last_became_peered":"2024-03-31T15:53:47.759278+0000","last_unstale":"2024-03-31T15:53:52.789642+0000","last_undegraded":"2024-03-31T15:53:52.789642+0000","last_fullsized":"2024-03-31T15:53:52.789642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:36.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247837+0000","last_change":"2024-03-31T15:53:47.759026+0000","last_active":"2024-03-31T15:53:53.247837+0000","last_peered":"2024-03-31T15:53:53.247837+0000","last_clean":"2024-03-31T15:53:53.247837+0000","last_became_active":"2024-03-31T15:53:47.758752+0000","last_became_peered":"2024-03-31T15:53:47.758752+0000","last_unstale":"2024-03-31T15:53:53.247837+0000","last_undegraded":"2024-03-31T15:53:53.247837+0000","last_fullsized":"2024-03-31T15:53:53.247837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:05:25.701168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790550+0000","last_change":"2024-03-31T15:53:47.762998+0000","last_active":"2024-03-31T15:53:52.790550+0000","last_peered":"2024-03-31T15:53:52.790550+0000","last_clean":"2024-03-31T15:53:52.790550+0000","last_became_active":"2024-03-31T15:53:47.762724+0000","last_became_peered":"2024-03-31T15:53:47.762724+0000","last_unstale":"2024-03-31T15:53:52.790550+0000","last_undegraded":"2024-03-31T15:53:52.790550+0000","last_fullsized":"2024-03-31T15:53:52.790550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359161+0000","last_change":"2024-03-31T15:53:47.761675+0000","last_active":"2024-03-31T15:53:53.359161+0000","last_peered":"2024-03-31T15:53:53.359161+0000","last_clean":"2024-03-31T15:53:53.359161+0000","last_became_active":"2024-03-31T15:53:47.757931+0000","last_became_peered":"2024-03-31T15:53:47.757931+0000","last_unstale":"2024-03-31T15:53:53.359161+0000","last_undegraded":"2024-03-31T15:53:53.359161+0000","last_fullsized":"2024-03-31T15:53:53.359161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:15.515856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306768+0000","last_change":"2024-03-31T15:53:47.770705+0000","last_active":"2024-03-31T15:53:53.306768+0000","last_peered":"2024-03-31T15:53:53.306768+0000","last_clean":"2024-03-31T15:53:53.306768+0000","last_became_active":"2024-03-31T15:53:47.770485+0000","last_became_peered":"2024-03-31T15:53:47.770485+0000","last_unstale":"2024-03-31T15:53:53.306768+0000","last_undegraded":"2024-03-31T15:53:53.306768+0000","last_fullsized":"2024-03-31T15:53:53.306768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792240+0000","last_change":"2024-03-31T15:53:47.761166+0000","last_active":"2024-03-31T15:53:52.792240+0000","last_peered":"2024-03-31T15:53:52.792240+0000","last_clean":"2024-03-31T15:53:52.792240+0000","last_became_active":"2024-03-31T15:53:47.760825+0000","last_became_peered":"2024-03-31T15:53:47.760825+0000","last_unstale":"2024-03-31T15:53:52.792240+0000","last_undegraded":"2024-03-31T15:53:52.792240+0000","last_fullsized":"2024-03-31T15:53:52.792240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:56.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528829+0000","last_change":"2024-03-31T15:53:47.765006+0000","last_active":"2024-03-31T15:53:53.528829+0000","last_peered":"2024-03-31T15:53:53.528829+0000","last_clean":"2024-03-31T15:53:53.528829+0000","last_became_active":"2024-03-31T15:53:47.760192+0000","last_became_peered":"2024-03-31T15:53:47.760192+0000","last_unstale":"2024-03-31T15:53:53.528829+0000","last_undegraded":"2024-03-31T15:53:53.528829+0000","last_fullsized":"2024-03-31T15:53:53.528829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:26.202073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790075+0000","last_change":"2024-03-31T15:53:47.765871+0000","last_active":"2024-03-31T15:53:52.790075+0000","last_peered":"2024-03-31T15:53:52.790075+0000","last_clean":"2024-03-31T15:53:52.790075+0000","last_became_active":"2024-03-31T15:53:47.764257+0000","last_became_peered":"2024-03-31T15:53:47.764257+0000","last_unstale":"2024-03-31T15:53:52.790075+0000","last_undegraded":"2024-03-31T15:53:52.790075+0000","last_fullsized":"2024-03-31T15:53:52.790075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:40.058921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789357+0000","last_change":"2024-03-31T15:53:47.764320+0000","last_active":"2024-03-31T15:53:52.789357+0000","last_peered":"2024-03-31T15:53:52.789357+0000","last_clean":"2024-03-31T15:53:52.789357+0000","last_became_active":"2024-03-31T15:53:47.764067+0000","last_became_peered":"2024-03-31T15:53:47.764067+0000","last_unstale":"2024-03-31T15:53:52.789357+0000","last_undegraded":"2024-03-31T15:53:52.789357+0000","last_fullsized":"2024-03-31T15:53:52.789357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920242+0000","last_change":"2024-03-31T15:53:47.755404+0000","last_active":"2024-03-31T15:53:52.920242+0000","last_peered":"2024-03-31T15:53:52.920242+0000","last_clean":"2024-03-31T15:53:52.920242+0000","last_became_active":"2024-03-31T15:53:47.755133+0000","last_became_peered":"2024-03-31T15:53:47.755133+0000","last_unstale":"2024-03-31T15:53:52.920242+0000","last_undegraded":"2024-03-31T15:53:52.920242+0000","last_fullsized":"2024-03-31T15:53:52.920242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359051+0000","last_change":"2024-03-31T15:53:47.751494+0000","last_active":"2024-03-31T15:53:53.359051+0000","last_peered":"2024-03-31T15:53:53.359051+0000","last_clean":"2024-03-31T15:53:53.359051+0000","last_became_active":"2024-03-31T15:53:47.751219+0000","last_became_peered":"2024-03-31T15:53:47.751219+0000","last_unstale":"2024-03-31T15:53:53.359051+0000","last_undegraded":"2024-03-31T15:53:53.359051+0000","last_fullsized":"2024-03-31T15:53:53.359051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247881+0000","last_change":"2024-03-31T15:53:47.766595+0000","last_active":"2024-03-31T15:53:53.247881+0000","last_peered":"2024-03-31T15:53:53.247881+0000","last_clean":"2024-03-31T15:53:53.247881+0000","last_became_active":"2024-03-31T15:53:47.766209+0000","last_became_peered":"2024-03-31T15:53:47.766209+0000","last_unstale":"2024-03-31T15:53:53.247881+0000","last_undegraded":"2024-03-31T15:53:53.247881+0000","last_fullsized":"2024-03-31T15:53:53.247881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:51.207164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793406+0000","last_change":"2024-03-31T15:53:47.762468+0000","last_active":"2024-03-31T15:53:52.793406+0000","last_peered":"2024-03-31T15:53:52.793406+0000","last_clean":"2024-03-31T15:53:52.793406+0000","last_became_active":"2024-03-31T15:53:47.762140+0000","last_became_peered":"2024-03-31T15:53:47.762140+0000","last_unstale":"2024-03-31T15:53:52.793406+0000","last_undegraded":"2024-03-31T15:53:52.793406+0000","last_fullsized":"2024-03-31T15:53:52.793406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:04.076153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306895+0000","last_change":"2024-03-31T15:53:47.774515+0000","last_active":"2024-03-31T15:53:53.306895+0000","last_peered":"2024-03-31T15:53:53.306895+0000","last_clean":"2024-03-31T15:53:53.306895+0000","last_became_active":"2024-03-31T15:53:47.774203+0000","last_became_peered":"2024-03-31T15:53:47.774203+0000","last_unstale":"2024-03-31T15:53:53.306895+0000","last_undegraded":"2024-03-31T15:53:53.306895+0000","last_fullsized":"2024-03-31T15:53:53.306895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307016+0000","last_change":"2024-03-31T15:53:47.775151+0000","last_active":"2024-03-31T15:53:53.307016+0000","last_peered":"2024-03-31T15:53:53.307016+0000","last_clean":"2024-03-31T15:53:53.307016+0000","last_became_active":"2024-03-31T15:53:47.774661+0000","last_became_peered":"2024-03-31T15:53:47.774661+0000","last_unstale":"2024-03-31T15:53:53.307016+0000","last_undegraded":"2024-03-31T15:53:53.307016+0000","last_fullsized":"2024-03-31T15:53:53.307016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792705+0000","last_change":"2024-03-31T15:53:47.772389+0000","last_active":"2024-03-31T15:53:52.792705+0000","last_peered":"2024-03-31T15:53:52.792705+0000","last_clean":"2024-03-31T15:53:52.792705+0000","last_became_active":"2024-03-31T15:53:47.772131+0000","last_became_peered":"2024-03-31T15:53:47.772131+0000","last_unstale":"2024-03-31T15:53:52.792705+0000","last_undegraded":"2024-03-31T15:53:52.792705+0000","last_fullsized":"2024-03-31T15:53:52.792705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:35.224566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247670+0000","last_change":"2024-03-31T15:53:47.772995+0000","last_active":"2024-03-31T15:53:53.247670+0000","last_peered":"2024-03-31T15:53:53.247670+0000","last_clean":"2024-03-31T15:53:53.247670+0000","last_became_active":"2024-03-31T15:53:47.772708+0000","last_became_peered":"2024-03-31T15:53:47.772708+0000","last_unstale":"2024-03-31T15:53:53.247670+0000","last_undegraded":"2024-03-31T15:53:53.247670+0000","last_fullsized":"2024-03-31T15:53:53.247670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:12:55.991650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791467+0000","last_change":"2024-03-31T15:53:47.766440+0000","last_active":"2024-03-31T15:53:52.791467+0000","last_peered":"2024-03-31T15:53:52.791467+0000","last_clean":"2024-03-31T15:53:52.791467+0000","last_became_active":"2024-03-31T15:53:47.766187+0000","last_became_peered":"2024-03-31T15:53:47.766187+0000","last_unstale":"2024-03-31T15:53:52.791467+0000","last_undegraded":"2024-03-31T15:53:52.791467+0000","last_fullsized":"2024-03-31T15:53:52.791467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:53.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790499+0000","last_change":"2024-03-31T15:53:47.778379+0000","last_active":"2024-03-31T15:53:52.790499+0000","last_peered":"2024-03-31T15:53:52.790499+0000","last_clean":"2024-03-31T15:53:52.790499+0000","last_became_active":"2024-03-31T15:53:47.778049+0000","last_became_peered":"2024-03-31T15:53:47.778049+0000","last_unstale":"2024-03-31T15:53:52.790499+0000","last_undegraded":"2024-03-31T15:53:52.790499+0000","last_fullsized":"2024-03-31T15:53:52.790499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791659+0000","last_change":"2024-03-31T15:53:47.761184+0000","last_active":"2024-03-31T15:53:52.791659+0000","last_peered":"2024-03-31T15:53:52.791659+0000","last_clean":"2024-03-31T15:53:52.791659+0000","last_became_active":"2024-03-31T15:53:47.760863+0000","last_became_peered":"2024-03-31T15:53:47.760863+0000","last_unstale":"2024-03-31T15:53:52.791659+0000","last_undegraded":"2024-03-31T15:53:52.791659+0000","last_fullsized":"2024-03-31T15:53:52.791659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:08:38.336677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789652+0000","last_change":"2024-03-31T15:53:47.766801+0000","last_active":"2024-03-31T15:53:52.789652+0000","last_peered":"2024-03-31T15:53:52.789652+0000","last_clean":"2024-03-31T15:53:52.789652+0000","last_became_active":"2024-03-31T15:53:47.766512+0000","last_became_peered":"2024-03-31T15:53:47.766512+0000","last_unstale":"2024-03-31T15:53:52.789652+0000","last_undegraded":"2024-03-31T15:53:52.789652+0000","last_fullsized":"2024-03-31T15:53:52.789652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:25.798405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359475+0000","last_change":"2024-03-31T15:53:47.756732+0000","last_active":"2024-03-31T15:53:53.359475+0000","last_peered":"2024-03-31T15:53:53.359475+0000","last_clean":"2024-03-31T15:53:53.359475+0000","last_became_active":"2024-03-31T15:53:47.756466+0000","last_became_peered":"2024-03-31T15:53:47.756466+0000","last_unstale":"2024-03-31T15:53:53.359475+0000","last_undegraded":"2024-03-31T15:53:53.359475+0000","last_fullsized":"2024-03-31T15:53:53.359475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921162+0000","last_change":"2024-03-31T15:53:47.763716+0000","last_active":"2024-03-31T15:53:52.921162+0000","last_peered":"2024-03-31T15:53:52.921162+0000","last_clean":"2024-03-31T15:53:52.921162+0000","last_became_active":"2024-03-31T15:53:47.763310+0000","last_became_peered":"2024-03-31T15:53:47.763310+0000","last_unstale":"2024-03-31T15:53:52.921162+0000","last_undegraded":"2024-03-31T15:53:52.921162+0000","last_fullsized":"2024-03-31T15:53:52.921162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792492+0000","last_change":"2024-03-31T15:53:47.765680+0000","last_active":"2024-03-31T15:53:52.792492+0000","last_peered":"2024-03-31T15:53:52.792492+0000","last_clean":"2024-03-31T15:53:52.792492+0000","last_became_active":"2024-03-31T15:53:47.765516+0000","last_became_peered":"2024-03-31T15:53:47.765516+0000","last_unstale":"2024-03-31T15:53:52.792492+0000","last_undegraded":"2024-03-31T15:53:52.792492+0000","last_fullsized":"2024-03-31T15:53:52.792492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:44:09.409693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":102303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":102143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":856064,"data_stored":693093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:54:13.379 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:54:13.380 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-31T15:54:13.380 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-31T15:54:13.380 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-31T15:54:13.380 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-31T15:54:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 Processor -- start 2024-03-31T15:54:13.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 -- start start 2024-03-31T15:54:13.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb14815c120 0x7fb14815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb1480cca30 0x7fb1480cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480d2b10 con 0x7fb1480d2680 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb148160bf0 con 0x7fb1480cca30 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb150800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb1480cd3a0 con 0x7fb14815c120 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb1480cca30 0x7fb1480cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb1480d2680 0x7fb1480b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33840/0 (socket says 172.21.15.90:33840) 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb1480cca30 0x7fb1480cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51184/0 (socket says 172.21.15.90:51184) 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14d400640 1 -- 172.21.15.90:0/783948576 learned_addr learned my addr 172.21.15.90:0/783948576 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:13.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14de00640 1 --2- 172.21.15.90:0/783948576 >> v2:172.21.15.102:3300/0 conn(0x7fb14815c120 0x7fb14815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 -- 172.21.15.90:0/783948576 >> v2:172.21.15.102:3300/0 conn(0x7fb14815c120 msgr2=0x7fb14815fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 --2- 172.21.15.90:0/783948576 >> v2:172.21.15.102:3300/0 conn(0x7fb14815c120 0x7fb14815fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 -- 172.21.15.90:0/783948576 >> v1:172.21.15.90:6789/0 conn(0x7fb1480d2680 legacy=0x7fb1480b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14e800640 1 -- 172.21.15.90:0/783948576 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb148160350 con 0x7fb1480cca30 2024-03-31T15:54:13.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.558+0000 7fb14ca00640 1 -- 172.21.15.90:0/783948576 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95589889 0 0) 0x7fb1480d2b10 con 0x7fb1480d2680 2024-03-31T15:54:13.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb14e800640 1 --2- 172.21.15.90:0/783948576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb1480cce60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1480d2d20 tx=0x7fb14401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6527ebd72b3ae127 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb14ca00640 1 -- 172.21.15.90:0/783948576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1440b2070 con 0x7fb1480cca30 2024-03-31T15:54:13.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb14ca00640 1 -- 172.21.15.90:0/783948576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1440a5930 con 0x7fb1480cca30 2024-03-31T15:54:13.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb14ca00640 1 -- 172.21.15.90:0/783948576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1440a5d00 con 0x7fb1480cca30 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 -- 172.21.15.90:0/783948576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 msgr2=0x7fb1480cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 --2- 172.21.15.90:0/783948576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb1480cce60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1480d2d20 tx=0x7fb14401c870 comp rx=0 tx=0).stop 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 -- 172.21.15.90:0/783948576 shutdown_connections 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 --2- 172.21.15.90:0/783948576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb1480cce60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 --2- 172.21.15.90:0/783948576 >> v2:172.21.15.102:3300/0 conn(0x7fb14815c120 0x7fb14815fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 -- 172.21.15.90:0/783948576 >> 172.21.15.90:0/783948576 conn(0x7fb14809aa40 msgr2=0x7fb14809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 -- 172.21.15.90:0/783948576 shutdown_connections 2024-03-31T15:54:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.562+0000 7fb150800640 1 -- 172.21.15.90:0/783948576 wait complete. 2024-03-31T15:54:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 Processor -- start 2024-03-31T15:54:13.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- start start 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1480d2680 0x7fb148155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb14815c120 0x7fb14806d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb14815c680 con 0x7fb1480cca30 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb14815c500 con 0x7fb1480d2680 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb1480b2df0 con 0x7fb14815c120 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51200/0 (socket says 172.21.15.90:51200) 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 -- 172.21.15.90:0/1186204675 learned_addr learned my addr 172.21.15.90:0/1186204675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 -- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb14815c120 msgr2=0x7fb14806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb14815c120 0x7fb14806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 -- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1480d2680 msgr2=0x7fb148155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1480d2680 0x7fb148155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 -- 172.21.15.90:0/1186204675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1480cbb50 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb14de00640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00e930 tx=0x7fb13c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c017070 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13c00ca00 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13c00cdb0 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1480cbe30 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.566+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1480c6a20 con 0x7fb1480cca30 2024-03-31T15:54:13.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1480686d0 con 0x7fb1480cca30 2024-03-31T15:54:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb13c01a070 con 0x7fb1480cca30 2024-03-31T15:54:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb142000640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 0x7fb12c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb13c087170 con 0x7fb1480cca30 2024-03-31T15:54:13.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb14d400640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 0x7fb12c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.570+0000 7fb14d400640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 0x7fb12c06a840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb148165780 tx=0x7fb138007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.574+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb13c057410 con 0x7fb1480cca30 2024-03-31T15:54:13.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.770+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb1480b1b60 con 0x7fb1480cca30 2024-03-31T15:54:13.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.774+0000 7fb142000640 1 -- 172.21.15.90:0/1186204675 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+168087 (secure 0 0 0) 0x7fb13c056d10 con 0x7fb1480cca30 2024-03-31T15:54:13.770 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:13.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.774+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 msgr2=0x7fb12c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.774+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 0x7fb12c06a840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb148165780 tx=0x7fb138007970 comp rx=0 tx=0).stop 2024-03-31T15:54:13.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.774+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 msgr2=0x7fb14815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.774+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00e930 tx=0x7fb13c00edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 shutdown_connections 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb12c068380 0x7fb12c06a840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb14815c120 0x7fb14806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1480d2680 0x7fb148155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 --2- 172.21.15.90:0/1186204675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb1480cca30 0x7fb14815a090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 >> 172.21.15.90:0/1186204675 conn(0x7fb14809aa40 msgr2=0x7fb14809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 shutdown_connections 2024-03-31T15:54:13.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.778+0000 7fb150800640 1 -- 172.21.15.90:0/1186204675 wait complete. 2024-03-31T15:54:13.790 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":158662636},{"type":"v1","addr":"172.21.15.102:6833","nonce":158662636}]},"active_addr":"172.21.15.102:6833/158662636","active_change":"2024-03-31T15:53:29.198423+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":89715519}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1788339680}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3734330082}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3719192422}]}]} 2024-03-31T15:54:13.793 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-31T15:54:13.793 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-31T15:54:13.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:13.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.934+0000 7f8accc00640 1 Processor -- start 2024-03-31T15:54:13.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.934+0000 7f8accc00640 1 -- start start 2024-03-31T15:54:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8accc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ac806c7b0 0x7f8ac806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8accc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8accc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac815f780 con 0x7f8ac815e1b0 2024-03-31T15:54:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8accc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8ac81622b0 con 0x7f8ac806c7b0 2024-03-31T15:54:13.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8accc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac806da60 con 0x7f8ac806d5f0 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ac806c7b0 0x7f8ac806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ac806c7b0 0x7f8ac806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51212/0 (socket says 172.21.15.90:51212) 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 -- 172.21.15.90:0/2465090022 learned_addr learned my addr 172.21.15.90:0/2465090022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8ac6800640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 -- 172.21.15.90:0/2465090022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 msgr2=0x7f8ac815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 -- 172.21.15.90:0/2465090022 >> v1:172.21.15.102:6789/0 conn(0x7f8ac815e1b0 legacy=0x7f8ac8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8abfe00640 1 -- 172.21.15.90:0/2465090022 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac815f1c0 con 0x7f8ac806c7b0 2024-03-31T15:54:13.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.938+0000 7f8ac6800640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:13.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8abfe00640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac806c7b0 0x7f8ac806d0b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009870 tx=0x7f8ab001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d81d2dad66f6100 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8ac5400640 1 -- 172.21.15.90:0/2465090022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00ab620 con 0x7f8ac806c7b0 2024-03-31T15:54:13.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8ac5400640 1 -- 172.21.15.90:0/2465090022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab00b0070 con 0x7f8ac806c7b0 2024-03-31T15:54:13.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8ac5400640 1 -- 172.21.15.90:0/2465090022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00ab9e0 con 0x7f8ac806c7b0 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- 172.21.15.90:0/2465090022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac806c7b0 msgr2=0x7f8ac806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac806c7b0 0x7f8ac806d0b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009870 tx=0x7f8ab001c460 comp rx=0 tx=0).stop 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- 172.21.15.90:0/2465090022 shutdown_connections 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 --2- 172.21.15.90:0/2465090022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac806c7b0 0x7f8ac806d0b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- 172.21.15.90:0/2465090022 >> 172.21.15.90:0/2465090022 conn(0x7f8ac809aa40 msgr2=0x7f8ac809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- 172.21.15.90:0/2465090022 shutdown_connections 2024-03-31T15:54:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- 172.21.15.90:0/2465090022 wait complete. 2024-03-31T15:54:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 Processor -- start 2024-03-31T15:54:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 -- start start 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.942+0000 7f8accc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac806c7b0 0x7f8ac80b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac81595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 0x7f8ac80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac80d2940 con 0x7f8ac815e1b0 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac80d27c0 con 0x7f8ac806c7b0 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac80d2ac0 con 0x7f8ac806d5f0 2024-03-31T15:54:13.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac81595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 0x7f8ac80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac806c7b0 0x7f8ac80b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abfe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac81595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49910/0 (socket says 172.21.15.90:49910) 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abfe00640 1 -- 172.21.15.90:0/1577535128 learned_addr learned my addr 172.21.15.90:0/1577535128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 -- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 msgr2=0x7f8ac81595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac81595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 -- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac806c7b0 msgr2=0x7f8ac80b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac806c7b0 0x7f8ac80b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 -- 172.21.15.90:0/1577535128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac80b5600 con 0x7f8ac815e1b0 2024-03-31T15:54:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8ac6800640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 0x7f8ac80b4b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000cc90 tx=0x7f8ac000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000f680 con 0x7f8ac815e1b0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac000f840 con 0x7f8ac815e1b0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000fc30 con 0x7f8ac815e1b0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac80b58e0 con 0x7f8ac815e1b0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac80aed20 con 0x7f8ac815e1b0 2024-03-31T15:54:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac80aeb10 con 0x7f8ac815e1b0 2024-03-31T15:54:13.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.946+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8ac000e3e0 con 0x7f8ac815e1b0 2024-03-31T15:54:13.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.950+0000 7f8abe000640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 0x7f8aa406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:13.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.950+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8ac0087940 con 0x7f8ac815e1b0 2024-03-31T15:54:13.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.950+0000 7f8ac5e00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 0x7f8aa406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:13.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.950+0000 7f8ac5e00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 0x7f8aa406a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8070500 tx=0x7f8ab8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:13.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:13.954+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8ac0014030 con 0x7f8ac815e1b0 2024-03-31T15:54:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.110+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ac80686d0 con 0x7f8ac815e1b0 2024-03-31T15:54:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8abe000640 1 -- 172.21.15.90:0/1577535128 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f8ac00574d0 con 0x7f8ac815e1b0 2024-03-31T15:54:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:14.108 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:14.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 msgr2=0x7f8aa406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 0x7f8aa406a870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8070500 tx=0x7f8ab8007e40 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 msgr2=0x7f8ac80b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 0x7f8ac80b4b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000cc90 tx=0x7f8ac000a650 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 shutdown_connections 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8aa40683b0 0x7f8aa406a870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ac815e1b0 0x7f8ac80b4b40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ac806d5f0 0x7f8ac81595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 --2- 172.21.15.90:0/1577535128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ac806c7b0 0x7f8ac80b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 >> 172.21.15.90:0/1577535128 conn(0x7f8ac809aa40 msgr2=0x7f8ac815fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 shutdown_connections 2024-03-31T15:54:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.114+0000 7f8accc00640 1 -- 172.21.15.90:0/1577535128 wait complete. 2024-03-31T15:54:14.129 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-31T15:54:14.130 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:14.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 Processor -- start 2024-03-31T15:54:14.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 -- start start 2024-03-31T15:54:14.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9000ccdb0 0x7ff9000b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9000b34f0 con 0x7ff9000cc9d0 2024-03-31T15:54:14.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff9000b3ab0 con 0x7ff9000ccdb0 2024-03-31T15:54:14.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff905e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff900141ba0 con 0x7ff90006b820 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9000ccdb0 0x7ff9000b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8ff400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff9000cc9d0 0x7ff900142030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45386/0 (socket says 172.21.15.90:45386) 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fea00640 1 -- 172.21.15.90:0/3665272933 learned_addr learned my addr 172.21.15.90:0/3665272933 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8ff400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49916/0 (socket says 172.21.15.90:49916) 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9000ccdb0 0x7ff9000b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:46932/0 (socket says 172.21.15.90:46932) 2024-03-31T15:54:14.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fd600640 1 -- 172.21.15.90:0/3665272933 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1655275021 0 0) 0x7ff9000b34f0 con 0x7ff9000cc9d0 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fd600640 1 -- 172.21.15.90:0/3665272933 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8e4003530 con 0x7ff9000cc9d0 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 -- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 msgr2=0x7ff90006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 -- 172.21.15.90:0/3665272933 >> v1:172.21.15.102:6789/0 conn(0x7ff9000cc9d0 legacy=0x7ff900142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8fe000640 1 -- 172.21.15.90:0/3665272933 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9000b37d0 con 0x7ff9000ccdb0 2024-03-31T15:54:14.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.322+0000 7ff8ff400640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:14.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff8fe000640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9000ccdb0 0x7ff9000b2b80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0097c0 tx=0x7ff8ec01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e269049f2fe0128 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff8fd600640 1 -- 172.21.15.90:0/3665272933 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0b0070 con 0x7ff9000ccdb0 2024-03-31T15:54:14.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff8fd600640 1 -- 172.21.15.90:0/3665272933 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ec0ab600 con 0x7ff9000ccdb0 2024-03-31T15:54:14.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff8fd600640 1 -- 172.21.15.90:0/3665272933 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0ab9d0 con 0x7ff9000ccdb0 2024-03-31T15:54:14.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 -- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9000ccdb0 msgr2=0x7ff9000b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9000ccdb0 0x7ff9000b2b80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec0097c0 tx=0x7ff8ec01c410 comp rx=0 tx=0).stop 2024-03-31T15:54:14.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 -- 172.21.15.90:0/3665272933 shutdown_connections 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff90006b820 0x7ff90006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 --2- 172.21.15.90:0/3665272933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9000ccdb0 0x7ff9000b2b80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 -- 172.21.15.90:0/3665272933 >> 172.21.15.90:0/3665272933 conn(0x7ff9000b1c80 msgr2=0x7ff9000b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 -- 172.21.15.90:0/3665272933 shutdown_connections 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.326+0000 7ff905e00640 1 -- 172.21.15.90:0/3665272933 wait complete. 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 Processor -- start 2024-03-31T15:54:14.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- start start 2024-03-31T15:54:14.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 0x7ff900152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff9000ccdb0 0x7ff90013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9000cf0c0 con 0x7ff9000cc9d0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff9000cef40 con 0x7ff90006b820 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff90015c620 con 0x7ff9000ccdb0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51228/0 (socket says 172.21.15.90:51228) 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 -- 172.21.15.90:0/965891214 learned_addr learned my addr 172.21.15.90:0/965891214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fea00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 0x7ff900152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8ff400640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff9000ccdb0 0x7ff90013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 -- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff9000ccdb0 msgr2=0x7ff90013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff9000ccdb0 0x7ff90013d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 -- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 msgr2=0x7ff900152b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 0x7ff900152b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 -- 172.21.15.90:0/965891214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff900153b20 con 0x7ff9000cc9d0 2024-03-31T15:54:14.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fea00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 0x7ff900152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8fe000640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec01a040 tx=0x7ff8ec0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0b0040 con 0x7ff9000cc9d0 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ec0bc070 con 0x7ff9000cc9d0 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff900153e00 con 0x7ff9000cc9d0 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec006e20 con 0x7ff9000cc9d0 2024-03-31T15:54:14.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.330+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9001543c0 con 0x7ff9000cc9d0 2024-03-31T15:54:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9000686d0 con 0x7ff9000cc9d0 2024-03-31T15:54:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff8ec0b3070 con 0x7ff9000cc9d0 2024-03-31T15:54:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff8f7400640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 0x7ff8e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff8ec121850 con 0x7ff9000cc9d0 2024-03-31T15:54:14.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff8fea00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 0x7ff8e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.334+0000 7ff8fea00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 0x7ff8e006a840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0005890 tx=0x7ff8f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.342+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff8ec016090 con 0x7ff9000cc9d0 2024-03-31T15:54:14.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.510+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff900167d90 con 0x7ff9000cc9d0 2024-03-31T15:54:14.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff8f7400640 1 -- 172.21.15.90:0/965891214 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7ff8ec0f1b40 con 0x7ff9000cc9d0 2024-03-31T15:54:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:14.508 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:14.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 msgr2=0x7ff8e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 0x7ff8e006a840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0005890 tx=0x7ff8f0009040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 msgr2=0x7ff90013d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec01a040 tx=0x7ff8ec0191b0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 shutdown_connections 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff8e0068380 0x7ff8e006a840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff9000ccdb0 0x7ff90013d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff9000cc9d0 0x7ff90013d130 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 --2- 172.21.15.90:0/965891214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff90006b820 0x7ff900152b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 >> 172.21.15.90:0/965891214 conn(0x7ff9000b1c80 msgr2=0x7ff9000b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 shutdown_connections 2024-03-31T15:54:14.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.514+0000 7ff905e00640 1 -- 172.21.15.90:0/965891214 wait complete. 2024-03-31T15:54:14.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-31T15:54:14.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-31T15:54:14.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-31T15:54:14.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-31T15:54:14.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-31T15:54:14.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-31T15:54:14.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-31T15:54:14.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-31T15:54:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 Processor -- start 2024-03-31T15:54:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 Processor -- start 2024-03-31T15:54:14.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 -- start start 2024-03-31T15:54:14.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f1d9a400640 1 Processor -- start 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2064072fd0 0x7f20640ca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2064166170 con 0x7f2064072bf0 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2064166d80 con 0x7f2064072fd0 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2068a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2064166450 con 0x7f20640cc740 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2061400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2064072fd0 0x7f20640ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49928/0 (socket says 172.21.15.90:49928) 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 -- 172.21.15.90:0/916267412 learned_addr learned my addr 172.21.15.90:0/916267412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 -- 172.21.15.90:0/916267412 >> v2:172.21.15.90:3300/0 conn(0x7f2064072fd0 msgr2=0x7f20640ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 --2- 172.21.15.90:0/916267412 >> v2:172.21.15.90:3300/0 conn(0x7f2064072fd0 0x7f20640ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f2062800640 1 -- 172.21.15.90:0/916267412 >> v1:172.21.15.90:6789/0 conn(0x7f2064072bf0 legacy=0x7f20640cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f2062800640 1 -- 172.21.15.90:0/916267412 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2064166730 con 0x7f20640cc740 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 -- start start 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66600715e0 0x7f66600ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666015a4e0 con 0x7f66600ce970 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f66600cc440 con 0x7f66600715e0 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f6664a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f66600cdfa0 con 0x7f666015a100 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f665f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f665f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49944/0 (socket says 172.21.15.90:49944) 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.782+0000 7f665f400640 1 -- 172.21.15.90:0/1762828869 learned_addr learned my addr 172.21.15.90:0/1762828869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d9a400640 1 -- start start 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d9a400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d9a400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d94163a30 con 0x7f1d940b5350 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d9a400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d9406f9b0 con 0x7f1d94162ff0 2024-03-31T15:54:14.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d9a400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d94163470 con 0x7f1d940b3710 2024-03-31T15:54:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d99000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d99000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49946/0 (socket says 172.21.15.90:49946) 2024-03-31T15:54:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f1d99000640 1 -- 172.21.15.90:0/4056641382 learned_addr learned my addr 172.21.15.90:0/4056641382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f665e000640 1 -- 172.21.15.90:0/1762828869 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 924512222 0 0) 0x7f666015a4e0 con 0x7f66600ce970 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f1d99000640 1 -- 172.21.15.90:0/4056641382 >> v1:172.21.15.102:6789/0 conn(0x7f1d94162ff0 legacy=0x7f1d9406f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f665ea00640 1 --2- 172.21.15.90:0/1762828869 >> v2:172.21.15.102:3300/0 conn(0x7f66600715e0 0x7f66600ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7f665e000640 1 -- 172.21.15.90:0/1762828869 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6644003530 con 0x7f66600ce970 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f665f400640 1 -- 172.21.15.90:0/1762828869 >> v2:172.21.15.102:3300/0 conn(0x7f66600715e0 msgr2=0x7f66600ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f665f400640 1 --2- 172.21.15.90:0/1762828869 >> v2:172.21.15.102:3300/0 conn(0x7f66600715e0 0x7f66600ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f665f400640 1 -- 172.21.15.90:0/1762828869 >> v1:172.21.15.90:6789/0 conn(0x7f66600ce970 legacy=0x7f66600ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f665f400640 1 -- 172.21.15.90:0/1762828869 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666015e870 con 0x7f666015a100 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 Processor -- start 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 -- start start 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fead0072fd0 0x7fead00ca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fead00cc740 0x7fead00ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fead0166170 con 0x7fead0072bf0 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fead00cd0e0 con 0x7fead0072fd0 2024-03-31T15:54:14.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead6a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fead0166d80 con 0x7fead00cc740 2024-03-31T15:54:14.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7feacfe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fead0072bf0 0x7fead00cbe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45404/0 (socket says 172.21.15.90:45404) 2024-03-31T15:54:14.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7feacfe00640 1 -- 172.21.15.90:0/1900292878 learned_addr learned my addr 172.21.15.90:0/1900292878 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:14.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7fead4a00640 1 --2- 172.21.15.90:0/1900292878 >> v2:172.21.15.102:3300/0 conn(0x7fead00cc740 0x7fead00ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.786+0000 7feacf400640 1 --2- 172.21.15.90:0/1900292878 >> v2:172.21.15.90:3300/0 conn(0x7fead0072fd0 0x7fead00ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f1d99000640 1 -- 172.21.15.90:0/4056641382 >> v1:172.21.15.90:6789/0 conn(0x7f1d940b5350 legacy=0x7f1d9406d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f1d99000640 1 -- 172.21.15.90:0/4056641382 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d94163750 con 0x7f1d940b3710 2024-03-31T15:54:14.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2062800640 1 --2- 172.21.15.90:0/916267412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2050009870 tx=0x7f205001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515f7b3b61cc88 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2060a00640 1 -- 172.21.15.90:0/916267412 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20500b0070 con 0x7f20640cc740 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2060a00640 1 -- 172.21.15.90:0/916267412 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20500ab600 con 0x7f20640cc740 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2060a00640 1 -- 172.21.15.90:0/916267412 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20500ab9d0 con 0x7f20640cc740 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 -- 172.21.15.90:0/916267412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 msgr2=0x7f20640ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 --2- 172.21.15.90:0/916267412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2050009870 tx=0x7f205001c490 comp rx=0 tx=0).stop 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 -- 172.21.15.90:0/916267412 shutdown_connections 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 --2- 172.21.15.90:0/916267412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640ccbd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 --2- 172.21.15.90:0/916267412 >> v2:172.21.15.90:3300/0 conn(0x7f2064072fd0 0x7f20640ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7f2068a00640 1 -- 172.21.15.90:0/916267412 >> 172.21.15.90:0/916267412 conn(0x7f20640b1880 msgr2=0x7f20640b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7fead4a00640 1 -- 172.21.15.90:0/1900292878 >> v2:172.21.15.90:3300/0 conn(0x7fead0072fd0 msgr2=0x7fead00ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7fead4a00640 1 --2- 172.21.15.90:0/1900292878 >> v2:172.21.15.90:3300/0 conn(0x7fead0072fd0 0x7fead00ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7fead4a00640 1 -- 172.21.15.90:0/1900292878 >> v1:172.21.15.102:6789/0 conn(0x7fead0072bf0 legacy=0x7fead00cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.790+0000 7fead4a00640 1 -- 172.21.15.90:0/1900292878 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead0166450 con 0x7fead00cc740 2024-03-31T15:54:14.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f2068a00640 1 -- 172.21.15.90:0/916267412 shutdown_connections 2024-03-31T15:54:14.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f665f400640 1 --2- 172.21.15.90:0/1762828869 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6650009870 tx=0x7f6650020940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0e8b918da33ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d99000640 1 --2- 172.21.15.90:0/4056641382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1d840098a0 tx=0x7f1d8401c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e365e368dedc18b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f665e000640 1 -- 172.21.15.90:0/1762828869 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66500b6070 con 0x7f666015a100 2024-03-31T15:54:14.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f665e000640 1 -- 172.21.15.90:0/1762828869 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66500a9810 con 0x7f666015a100 2024-03-31T15:54:14.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f665e000640 1 -- 172.21.15.90:0/1762828869 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66500a9be0 con 0x7f666015a100 2024-03-31T15:54:14.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7fea81800640 1 Processor -- start 2024-03-31T15:54:14.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d93400640 1 -- 172.21.15.90:0/4056641382 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d840aa670 con 0x7f1d940b3710 2024-03-31T15:54:14.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d93400640 1 -- 172.21.15.90:0/4056641382 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d840af070 con 0x7f1d940b3710 2024-03-31T15:54:14.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d93400640 1 -- 172.21.15.90:0/4056641382 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d840aaa30 con 0x7f1d940b3710 2024-03-31T15:54:14.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f6664a00640 1 -- 172.21.15.90:0/1762828869 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 msgr2=0x7f66600710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f6664a00640 1 --2- 172.21.15.90:0/1762828869 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6650009870 tx=0x7f6650020940 comp rx=0 tx=0).stop 2024-03-31T15:54:14.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d9a400640 1 -- 172.21.15.90:0/4056641382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 msgr2=0x7f1d94162ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f1d9a400640 1 --2- 172.21.15.90:0/4056641382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1d840098a0 tx=0x7f1d8401c530 comp rx=0 tx=0).stop 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f1d9a400640 1 -- 172.21.15.90:0/4056641382 shutdown_connections 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f1d9a400640 1 --2- 172.21.15.90:0/4056641382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b3710 0x7f1d94162ab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f1d9a400640 1 -- 172.21.15.90:0/4056641382 >> 172.21.15.90:0/4056641382 conn(0x7f1d940b1da0 msgr2=0x7f1d940b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f1d9a400640 1 -- 172.21.15.90:0/4056641382 shutdown_connections 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f1d9a400640 1 -- 172.21.15.90:0/4056641382 wait complete. 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7f2068a00640 1 -- 172.21.15.90:0/916267412 wait complete. 2024-03-31T15:54:14.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f2068a00640 1 Processor -- start 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.794+0000 7fea81800640 1 -- start start 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea81800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fea7c0cce50 0x7fea7c161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea81800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c0cc510 con 0x7fea7c072c30 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 -- 172.21.15.90:0/1762828869 shutdown_connections 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 --2- 172.21.15.90:0/1762828869 >> v2:172.21.15.102:3300/0 conn(0x7f66600715e0 0x7f66600ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 --2- 172.21.15.90:0/1762828869 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f666015a100 0x7f66600710a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 -- 172.21.15.90:0/1762828869 >> 172.21.15.90:0/1762828869 conn(0x7f66600b16a0 msgr2=0x7f66600b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea81800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c166310 con 0x7fea7c06ed80 2024-03-31T15:54:14.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea81800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fea7c1665f0 con 0x7fea7c0cce50 2024-03-31T15:54:14.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fead4a00640 1 --2- 172.21.15.90:0/1900292878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead00ccba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fead00b5e70 tx=0x7feac801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb7be6ee7903a3f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7aa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fea7c072c30 0x7fea7c0cbe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33870/0 (socket says 172.21.15.90:33870) 2024-03-31T15:54:14.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7aa00640 1 -- 172.21.15.90:0/2779625782 learned_addr learned my addr 172.21.15.90:0/2779625782 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:14.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7b400640 1 --2- 172.21.15.90:0/2779625782 >> v2:172.21.15.90:3300/0 conn(0x7fea7c0cce50 0x7fea7c161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 -- 172.21.15.90:0/1762828869 shutdown_connections 2024-03-31T15:54:14.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7f6664a00640 1 -- 172.21.15.90:0/1762828869 wait complete. 2024-03-31T15:54:14.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7feacea00640 1 -- 172.21.15.90:0/1900292878 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac80b2070 con 0x7fead00cc740 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7feacea00640 1 -- 172.21.15.90:0/1900292878 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac80a5850 con 0x7fead00cc740 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7feacea00640 1 -- 172.21.15.90:0/1900292878 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac80a5c20 con 0x7fead00cc740 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7b400640 1 -- 172.21.15.90:0/2779625782 >> v1:172.21.15.102:6789/0 conn(0x7fea7c06ed80 legacy=0x7fea7c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7b400640 1 -- 172.21.15.90:0/2779625782 >> v1:172.21.15.90:6789/0 conn(0x7fea7c072c30 legacy=0x7fea7c0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.798+0000 7fea7b400640 1 -- 172.21.15.90:0/2779625782 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c1668d0 con 0x7fea7c0cce50 2024-03-31T15:54:14.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 -- 172.21.15.90:0/1900292878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 msgr2=0x7fead00ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 --2- 172.21.15.90:0/1900292878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead00ccba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fead00b5e70 tx=0x7feac801c8c0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 Processor -- start 2024-03-31T15:54:14.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 -- start start 2024-03-31T15:54:14.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 Processor -- start 2024-03-31T15:54:14.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 -- 172.21.15.90:0/1900292878 shutdown_connections 2024-03-31T15:54:14.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 --2- 172.21.15.90:0/1900292878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead00ccba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 --2- 172.21.15.90:0/1900292878 >> v2:172.21.15.90:3300/0 conn(0x7fead0072fd0 0x7fead00ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 -- 172.21.15.90:0/1900292878 >> 172.21.15.90:0/1900292878 conn(0x7fead00b1880 msgr2=0x7fead00b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 -- 172.21.15.90:0/1900292878 shutdown_connections 2024-03-31T15:54:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7fead6a00640 1 -- 172.21.15.90:0/1900292878 wait complete. 2024-03-31T15:54:14.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f66600715e0 0x7f6660154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 0x7f66600c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f666015a100 0x7f66600c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f666015e620 con 0x7f666015a100 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f666015e4a0 con 0x7f66600ce970 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f6664a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f66600ca830 con 0x7f66600715e0 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f666015a100 0x7f66600c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 0x7f66600c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 0x7f66600c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:46976/0 (socket says 172.21.15.90:46976) 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 -- 172.21.15.90:0/2676223699 learned_addr learned my addr 172.21.15.90:0/2676223699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665f400640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f66600715e0 0x7f6660154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 -- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f66600715e0 msgr2=0x7f6660154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 -- start start 2024-03-31T15:54:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d940b3710 0x7f1d94154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 0x7f1d94159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d94162ff0 0x7f1d941fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d940b3c70 con 0x7f1d940b3710 2024-03-31T15:54:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1d940b3af0 con 0x7f1d94162ff0 2024-03-31T15:54:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f1d9a400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9406dbd0 con 0x7f1d940b5350 2024-03-31T15:54:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 0x7f1d94159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f2068a00640 1 -- start start 2024-03-31T15:54:14.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f66600715e0 0x7f6660154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 -- 172.21.15.90:0/2676223699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f666015a100 msgr2=0x7f66600c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f666015a100 0x7f66600c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.802+0000 7f665ea00640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66600c7530 con 0x7f66600ce970 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f665fe00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f666015a100 0x7f66600c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 0x7f1d94159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:49970/0 (socket says 172.21.15.90:49970) 2024-03-31T15:54:14.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 -- 172.21.15.90:0/2548156018 learned_addr learned my addr 172.21.15.90:0/2548156018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f665ea00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 0x7f66600c94d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f664c00dba0 tx=0x7f664c00a5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d99a00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d94162ff0 0x7f1d941fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d99000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d940b3710 0x7f1d94154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 -- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d94162ff0 msgr2=0x7f1d941fe8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d94162ff0 0x7f1d941fe8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 -- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d940b3710 msgr2=0x7f1d94154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d940b3710 0x7f1d94154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9415a000 con 0x7f1d940b5350 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d99a00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d94162ff0 0x7f1d941fe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:14.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d99000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d940b3710 0x7f1d94154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 Processor -- start 2024-03-31T15:54:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c0173f0 con 0x7f66600ce970 2024-03-31T15:54:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66600c7810 con 0x7f66600ce970 2024-03-31T15:54:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66600c7dd0 con 0x7f66600ce970 2024-03-31T15:54:14.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664c017590 con 0x7f66600ce970 2024-03-31T15:54:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c0173f0 con 0x7f66600ce970 2024-03-31T15:54:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6638001040 con 0x7f66600ce970 2024-03-31T15:54:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 -- start start 2024-03-31T15:54:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d93e00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 0x7f1d94159790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c370 tx=0x7f1d8000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fea7b400640 1 --2- 172.21.15.90:0/2779625782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c0cce50 0x7fea7c161c10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea74009d90 tx=0x7fea7401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed3633834e2101c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d800052d0 con 0x7f1d940b5350 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d80005470 con 0x7f1d940b5350 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80007260 con 0x7f1d940b5350 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fead0072bf0 0x7fead00b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead0072fd0 0x7fead0155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead0147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fead01625a0 con 0x7fead0072fd0 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fead0162420 con 0x7fead00cc740 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.806+0000 7fead6a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fead0162720 con 0x7fead0072bf0 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead0147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fea79600640 1 -- 172.21.15.90:0/2779625782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740b2070 con 0x7fea7c0cce50 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fea79600640 1 -- 172.21.15.90:0/2779625782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea740a5a00 con 0x7fea7c0cce50 2024-03-31T15:54:14.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fea79600640 1 -- 172.21.15.90:0/2779625782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea740a5dd0 con 0x7fea7c0cce50 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead0147930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:46990/0 (socket says 172.21.15.90:46990) 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 -- 172.21.15.90:0/3892281990 learned_addr learned my addr 172.21.15.90:0/3892281990 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feacfe00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fead0072bf0 0x7fead00b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feacf400640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead0072fd0 0x7fead0155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f1d9a400640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9415a340 con 0x7f1d940b5350 2024-03-31T15:54:14.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f1d9a400640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9415ab80 con 0x7f1d940b5350 2024-03-31T15:54:14.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 -- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fead0072bf0 msgr2=0x7fead00b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fead0072bf0 0x7fead00b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 -- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead0072fd0 msgr2=0x7fead0155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fead0072fd0 0x7fead0155a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f664c017740 con 0x7f66600ce970 2024-03-31T15:54:14.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2064072bf0 0x7f206415c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead00b58d0 con 0x7fead00cc740 2024-03-31T15:54:14.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feacfe00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fead0072bf0 0x7fead00b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2061e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2064072bf0 0x7f206415c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2061e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2064072bf0 0x7f206415c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47004/0 (socket says 172.21.15.90:47004) 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2061e00640 1 -- 172.21.15.90:0/1180812375 learned_addr learned my addr 172.21.15.90:0/1180812375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2064072fd0 0x7f20640b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead4a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead0147930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feac80194d0 tx=0x7feac8019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6640068780 0x7f664006ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f664c00b040 con 0x7f66600ce970 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f20641627f0 con 0x7f2064072fd0 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2064162670 con 0x7f2064072bf0 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac80bd030 con 0x7fead00cc740 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac80b3040 con 0x7fead00cc740 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac80b2070 con 0x7fead00cc740 2024-03-31T15:54:14.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2064162970 con 0x7f20640cc740 2024-03-31T15:54:14.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead00b42f0 con 0x7fead00cc740 2024-03-31T15:54:14.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1d94068d00 con 0x7f1d940b5350 2024-03-31T15:54:14.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead00b48b0 con 0x7fead00cc740 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7f6640071df0 0x7f6640074230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f664c001d40 con 0x7f6640071df0 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f664c0877a0 con 0x7f66600ce970 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 -- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2064072bf0 msgr2=0x7f206415c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2064072bf0 0x7f206415c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 -- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2064072fd0 msgr2=0x7f20640b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2064072fd0 0x7f20640b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.810+0000 7f2062800640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20640b4500 con 0x7f20640cc740 2024-03-31T15:54:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 -- 172.21.15.90:0/2779625782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c0cce50 msgr2=0x7fea7c161c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 --2- 172.21.15.90:0/2779625782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c0cce50 0x7fea7c161c10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea74009d90 tx=0x7fea7401c940 comp rx=0 tx=0).stop 2024-03-31T15:54:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feaa8001040 con 0x7fead00cc740 2024-03-31T15:54:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f665fe00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7f6640071df0 0x7f6640074230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f665f400640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6640068780 0x7f664006ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 -- 172.21.15.90:0/2779625782 shutdown_connections 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 --2- 172.21.15.90:0/2779625782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c0cce50 0x7fea7c161c10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 -- 172.21.15.90:0/2779625782 >> 172.21.15.90:0/2779625782 conn(0x7fea7c0b1a50 msgr2=0x7fea7c0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 -- 172.21.15.90:0/2779625782 shutdown_connections 2024-03-31T15:54:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7fea81800640 1 -- 172.21.15.90:0/2779625782 wait complete. 2024-03-31T15:54:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f665fe00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7f6640071df0 0x7f6640074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f2062800640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640c9550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20500ab000 tx=0x7f2050019230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d80007400 con 0x7f1d940b5350 2024-03-31T15:54:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== osd.4 v2:172.21.15.102:6800/101321808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f664c001d40 con 0x7f6640071df0 2024-03-31T15:54:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20500bb020 con 0x7f20640cc740 2024-03-31T15:54:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20500b1040 con 0x7f20640cc740 2024-03-31T15:54:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20500b0070 con 0x7f20640cc740 2024-03-31T15:54:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7f665f400640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6640068780 0x7f664006ac40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66500b3000 tx=0x7f6650008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea81800640 1 Processor -- start 2024-03-31T15:54:14.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea81800640 1 -- start start 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d90c00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d74068450 0x7f1d7406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1d80086370 con 0x7f1d940b5350 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d90c00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7f1d74071bd0 0x7f1d74074010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d80009e50 con 0x7f1d74071bd0 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d80086760 con 0x7f1d940b5350 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d99000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d74068450 0x7f1d7406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.814+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feac80c0070 con 0x7fead00cc740 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feac7e00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feab0068380 0x7feab006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feac8002020 con 0x7fead00cc740 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feac7e00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7feab0071b00 0x7feab0073f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feac80bee70 con 0x7feab0071b00 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7feac8128050 con 0x7fead00cc740 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20640b47e0 con 0x7f20640cc740 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20640b4da0 con 0x7f20640cc740 2024-03-31T15:54:14.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feacfe00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feab0068380 0x7feab006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d99a00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7f1d74071bd0 0x7f1d74074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f203c001040 con 0x7f20640cc740 2024-03-31T15:54:14.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7f1d99000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d74068450 0x7f1d7406a910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1d840ae000 tx=0x7f1d840a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea81800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 0x7fea7c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea81800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 0x7fea7c162f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7feacf400640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7feab0071b00 0x7feab0073f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea7aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 0x7fea7c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.818+0000 7fea7aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 0x7fea7c1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51304/0 (socket says 172.21.15.90:51304) 2024-03-31T15:54:14.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fea7c0cce50 0x7fea7c163450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 0x7fea7c162f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7f1d99a00640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7f1d74071bd0 0x7f1d74074010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 0x7fea7c162f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47016/0 (socket says 172.21.15.90:47016) 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c170680 con 0x7fea7c06ed80 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c170500 con 0x7fea7c072c30 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fea7c170800 con 0x7fea7c0cce50 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7aa00640 1 -- 172.21.15.90:0/3221324341 learned_addr learned my addr 172.21.15.90:0/3221324341 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 -- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fea7c0cce50 msgr2=0x7fea7c163450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fea7c0cce50 0x7fea7c163450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 -- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 msgr2=0x7fea7c1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 0x7fea7c1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c0b3640 con 0x7fea7c072c30 2024-03-31T15:54:14.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f20500be020 con 0x7f20640cc740 2024-03-31T15:54:14.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7aa00640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fea7c06ed80 0x7fea7c1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:14.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea7a000640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 0x7fea7c162f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f70 tx=0x7fea6800c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== osd.0 v2:172.21.15.90:6809/2322588474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1d80009e50 con 0x7f1d74071bd0 2024-03-31T15:54:14.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68017070 con 0x7fea7c072c30 2024-03-31T15:54:14.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7feacfe00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feab0068380 0x7feab006a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feac0000f70 tx=0x7feac0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7feacf400640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7feab0071b00 0x7feab0073f40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7c0b3920 con 0x7fea7c072c30 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.822+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7c0b3ee0 con 0x7fea7c072c30 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea680052d0 con 0x7fea7c072c30 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea680054c0 con 0x7fea7c072c30 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fea54001040 con 0x7fea7c072c30 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f205a000640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2044068450 0x7f204406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2050121370 con 0x7f20640cc740 2024-03-31T15:54:14.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f205a000640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f2044071bd0 0x7f2044074010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20440746e0 con 0x7f2044071bd0 2024-03-31T15:54:14.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2050121760 con 0x7f20640cc740 2024-03-31T15:54:14.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f2061e00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2044068450 0x7f204406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== osd.3 v2:172.21.15.90:6808/4045683431 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feac80bee70 con 0x7feab0071b00 2024-03-31T15:54:14.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.826+0000 7f2061400640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f2044071bd0 0x7f2044074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fea6801a070 con 0x7fea7c072c30 2024-03-31T15:54:14.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7f2061400640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f2044071bd0 0x7f2044074010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7f2061e00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2044068450 0x7f204406a910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f205c009fd0 tx=0x7f205c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea72a00640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fea5c068380 0x7fea5c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fea68085e80 con 0x7fea7c072c30 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea7aa00640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fea5c068380 0x7fea5c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea7aa00640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fea5c068380 0x7fea5c06a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea6c005890 tx=0x7fea6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea72a00640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7fea5c071a80 0x7fea5c073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea5c074590 con 0x7fea5c071a80 2024-03-31T15:54:14.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.830+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== osd.1 v2:172.21.15.90:6824/3383585142 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f20440746e0 con 0x7f2044071bd0 2024-03-31T15:54:14.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.834+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fea68051d40 con 0x7fea7c072c30 2024-03-31T15:54:14.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.834+0000 7fea7b400640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7fea5c071a80 0x7fea5c073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.834+0000 7fea7b400640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7fea5c071a80 0x7fea5c073ec0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.834+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== osd.2 v2:172.21.15.90:6800/347246527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea5c074590 con 0x7fea5c071a80 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.838+0000 7f2d20e00640 1 Processor -- start 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.838+0000 7f2d20e00640 1 -- start start 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.838+0000 7f2d20e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2d1c067330 0x7f2d1c0b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d20e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d1c0b1560 con 0x7f2d1c0cd530 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d20e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d1c0b5150 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d20e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2d1c161c50 con 0x7f2d1c067330 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2d1c06b5f0 0x7f2d1c06ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45426/0 (socket says 172.21.15.90:45426) 2024-03-31T15:54:14.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1be00640 1 -- 172.21.15.90:0/2335130231 learned_addr learned my addr 172.21.15.90:0/2335130231 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/2335130231 >> v2:172.21.15.90:3300/0 conn(0x7f2d1c067330 0x7f2d1c0b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274285129 0 0) 0x7f2d1c0b5150 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d00003530 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4224767264 0 0) 0x7f2d00003530 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 >> v2:172.21.15.90:3300/0 conn(0x7f2d1c067330 msgr2=0x7f2d1c0b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 --2- 172.21.15.90:0/2335130231 >> v2:172.21.15.90:3300/0 conn(0x7f2d1c067330 0x7f2d1c0b5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 >> v1:172.21.15.90:6789/0 conn(0x7f2d1c0cd530 legacy=0x7f2d1c0b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c0b5430 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f2d14001a00 con 0x7f2d1c06b5f0 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 >> v1:172.21.15.102:6789/0 conn(0x7f2d1c06b5f0 legacy=0x7f2d1c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d000039c0 0x7f2d00003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 0x7f2d00004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d000051d0 0x7f2d0000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c161c50 con 0x7f2d000039c0 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0000c170 con 0x7f2d00004810 2024-03-31T15:54:14.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0000c370 con 0x7f2d000051d0 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 0x7f2d00004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1be00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d000051d0 0x7f2d0000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1b400640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d000039c0 0x7f2d00003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 -- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d000051d0 msgr2=0x7f2d0000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d000051d0 0x7f2d0000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 -- 172.21.15.90:0/2335130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d000039c0 msgr2=0x7f2d00003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d000039c0 0x7f2d00003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1aa00640 1 -- 172.21.15.90:0/2335130231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c0b5430 con 0x7f2d00004810 2024-03-31T15:54:14.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.842+0000 7f2d1be00640 1 -- 172.21.15.90:0/2335130231 reap_dead start 2024-03-31T15:54:14.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.846+0000 7fb34de00640 1 Processor -- start 2024-03-31T15:54:14.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.846+0000 7fb34de00640 1 -- start start 2024-03-31T15:54:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.846+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 --> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6638002de0 con 0x7f6640071df0 2024-03-31T15:54:14.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb34de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb3480d2de0 0x7fb3480d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb34de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 0x7fb3480b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb34de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3480d4e30 con 0x7fb348161cc0 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb34de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb3480d4be0 con 0x7fb3480d2de0 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb34de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb3480b1430 con 0x7fb3480b2fa0 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7f6657400640 1 -- 172.21.15.90:0/2676223699 <== osd.4 v2:172.21.15.102:6800/101321808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6638002de0 con 0x7f6640071df0 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb346a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb348161cc0 0x7fb3480d26d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33884/0 (socket says 172.21.15.90:33884) 2024-03-31T15:54:14.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb346a00640 1 -- 172.21.15.90:0/3044658808 learned_addr learned my addr 172.21.15.90:0/3044658808 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:14.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 0x7f2d00004c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c009670 tx=0x7f2d0c0252a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87d4df5931d57199 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c0ba070 con 0x7f2d00004810 2024-03-31T15:54:14.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb347400640 1 --2- 172.21.15.90:0/3044658808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 0x7fb3480b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0c0adc40 con 0x7f2d00004810 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb347400640 1 -- 172.21.15.90:0/3044658808 >> v2:172.21.15.90:3300/0 conn(0x7fb3480d2de0 msgr2=0x7fb3480d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb347400640 1 --2- 172.21.15.90:0/3044658808 >> v2:172.21.15.90:3300/0 conn(0x7fb3480d2de0 0x7fb3480d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb347400640 1 -- 172.21.15.90:0/3044658808 >> v1:172.21.15.90:6789/0 conn(0x7fb348161cc0 legacy=0x7fb3480d26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7fb347400640 1 -- 172.21.15.90:0/3044658808 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3480b0de0 con 0x7fb3480b2fa0 2024-03-31T15:54:14.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.850+0000 7f2d1a000640 1 -- 172.21.15.90:0/2335130231 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c0096b0 con 0x7f2d00004810 2024-03-31T15:54:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7f6640071df0 msgr2=0x7f6640074230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7f6640071df0 0x7f6640074230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6640068780 msgr2=0x7f664006ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6640068780 0x7f664006ac40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f66500b3000 tx=0x7f6650008040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 msgr2=0x7f66600c94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f6664a00640 1 --2- 172.21.15.90:0/2676223699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66600ce970 0x7f66600c94d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f664c00dba0 tx=0x7f664c00a5a0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f665fe00640 1 -- 172.21.15.90:0/2676223699 reap_dead start 2024-03-31T15:54:14.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb347400640 1 --2- 172.21.15.90:0/3044658808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 0x7fb3480b18c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb340009d90 tx=0x7fb34001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5b2412e9ce45a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f2d20e00640 1 -- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 msgr2=0x7f2d00004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7f2d20e00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 0x7f2d00004c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c009670 tx=0x7f2d0c0252a0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb345600640 1 -- 172.21.15.90:0/3044658808 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400b2070 con 0x7fb3480b2fa0 2024-03-31T15:54:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb345600640 1 -- 172.21.15.90:0/3044658808 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3400a5910 con 0x7fb3480b2fa0 2024-03-31T15:54:14.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb345600640 1 -- 172.21.15.90:0/3044658808 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3400a5ce0 con 0x7fb3480b2fa0 2024-03-31T15:54:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 -- 172.21.15.90:0/3044658808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 msgr2=0x7fb3480b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 --2- 172.21.15.90:0/3044658808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 0x7fb3480b18c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb340009d90 tx=0x7fb34001c890 comp rx=0 tx=0).stop 2024-03-31T15:54:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 -- 172.21.15.90:0/3044658808 shutdown_connections 2024-03-31T15:54:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 --2- 172.21.15.90:0/3044658808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480b2fa0 0x7fb3480b18c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 --2- 172.21.15.90:0/3044658808 >> v2:172.21.15.90:3300/0 conn(0x7fb3480d2de0 0x7fb3480d1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 -- 172.21.15.90:0/3044658808 >> 172.21.15.90:0/3044658808 conn(0x7fb3480d40f0 msgr2=0x7fb3480d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 -- 172.21.15.90:0/3044658808 shutdown_connections 2024-03-31T15:54:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 shutdown_connections 2024-03-31T15:54:14.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 >> 172.21.15.90:0/2676223699 conn(0x7f66600b16a0 msgr2=0x7f66600c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f2d20e00640 1 -- 172.21.15.90:0/2335130231 shutdown_connections 2024-03-31T15:54:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f2d20e00640 1 --2- 172.21.15.90:0/2335130231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d00004810 0x7f2d00004c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f2d20e00640 1 -- 172.21.15.90:0/2335130231 >> 172.21.15.90:0/2335130231 conn(0x7f2d1c0d03e0 msgr2=0x7f2d1c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.854+0000 7fb34de00640 1 -- 172.21.15.90:0/3044658808 wait complete. 2024-03-31T15:54:14.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 --> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1d94155930 con 0x7f1d74071bd0 2024-03-31T15:54:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f1d90c00640 1 -- 172.21.15.90:0/2548156018 <== osd.0 v2:172.21.15.90:6809/2322588474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1d94155930 con 0x7f1d74071bd0 2024-03-31T15:54:14.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f2d20e00640 1 -- 172.21.15.90:0/2335130231 shutdown_connections 2024-03-31T15:54:14.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7f2d20e00640 1 -- 172.21.15.90:0/2335130231 wait complete. 2024-03-31T15:54:14.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7fb34de00640 1 Processor -- start 2024-03-31T15:54:14.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f2d20e00640 1 Processor -- start 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 shutdown_connections 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f6664a00640 1 -- 172.21.15.90:0/2676223699 wait complete. 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.858+0000 7fb34de00640 1 -- start start 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3480b2fa0 0x7fb34814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 0x7fb348159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7f1d74071bd0 msgr2=0x7f1d74074010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7f1d74071bd0 0x7f1d74074010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 0x7fb348159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3480b2a20 con 0x7fb3480b2fa0 2024-03-31T15:54:14.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb3480b28a0 con 0x7fb348161cc0 2024-03-31T15:54:14.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb34de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb3480b2ba0 con 0x7fb3480d2de0 2024-03-31T15:54:14.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f2d20e00640 1 -- start start 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 0x7fb348159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb347400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 0x7fb348159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 0x7fb348159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50004/0 (socket says 172.21.15.90:50004) 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb347400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 0x7fb348159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47036/0 (socket says 172.21.15.90:47036) 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 -- 172.21.15.90:0/1164036685 learned_addr learned my addr 172.21.15.90:0/1164036685 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346a00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3480b2fa0 0x7fb34814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 --> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f203c002de0 con 0x7f2044071bd0 2024-03-31T15:54:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d74068450 msgr2=0x7f1d7406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d74068450 0x7f1d7406a910 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1d840ae000 tx=0x7f1d840a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:14.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 -- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 msgr2=0x7fb348159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 msgr2=0x7f1d94159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 --2- 172.21.15.90:0/2548156018 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d940b5350 0x7f1d94159790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c370 tx=0x7f1d8000c830 comp rx=0 tx=0).stop 2024-03-31T15:54:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d99a00640 1 -- 172.21.15.90:0/2548156018 reap_dead start 2024-03-31T15:54:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 shutdown_connections 2024-03-31T15:54:14.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f205a000640 1 -- 172.21.15.90:0/1180812375 <== osd.1 v2:172.21.15.90:6824/3383585142 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f203c002de0 con 0x7f2044071bd0 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 >> 172.21.15.90:0/2548156018 conn(0x7f1d940b1da0 msgr2=0x7f1d94157760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 0x7fb348159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 -- 172.21.15.90:0/1164036685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3480b2fa0 msgr2=0x7fb34814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3480b2fa0 0x7fb34814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3481fb1a0 con 0x7fb3480d2de0 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346a00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3480b2fa0 0x7fb34814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb347400640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb348161cc0 0x7fb348159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:14.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7fb346000640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 0x7fb348159120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb33400b410 tx=0x7fb33400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 shutdown_connections 2024-03-31T15:54:14.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f1d8a000640 1 -- 172.21.15.90:0/2548156018 wait complete. 2024-03-31T15:54:14.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f2d20e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d00004810 0x7f2d1c1f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.862+0000 7f2d20e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 0x7f2d1c0c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d1c0c6460 0x7f2d1c0cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d00004810 0x7f2d1c1f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 0x7f2d1c0c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 0x7f2d1c0c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47040/0 (socket says 172.21.15.90:47040) 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 -- 172.21.15.90:0/489706889 learned_addr learned my addr 172.21.15.90:0/489706889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334015020 con 0x7fb3480d2de0 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3480cbbd0 con 0x7fb3480d2de0 2024-03-31T15:54:14.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3480cbe90 con 0x7fb3480d2de0 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1be00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d1c0c6460 0x7f2d1c0cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0ce940 con 0x7f2d1c0c6460 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0ce7c0 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c0ceac0 con 0x7f2d00004810 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb334011830 con 0x7fb3480d2de0 2024-03-31T15:54:14.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334011c20 con 0x7fb3480d2de0 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 -- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d00004810 msgr2=0x7f2d1c1f68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2d00004810 0x7f2d1c1f68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 -- 172.21.15.90:0/489706889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d1c0c6460 msgr2=0x7f2d1c0cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2d1c0c6460 0x7f2d1c0cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c1f6e20 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb320001040 con 0x7fb3480d2de0 2024-03-31T15:54:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d1aa00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 0x7f2d1c0c5f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0b8040 tx=0x7f2d0c022040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2b64000640 1 Processor -- start 2024-03-31T15:54:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c0ba040 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0c000c40 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c0b59c0 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2b64000640 1 -- start start 2024-03-31T15:54:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c09a860 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d20e00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c09ae20 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 --> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feaa8002de0 con 0x7feab0071b00 2024-03-31T15:54:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d1c068d00 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f2044071bd0 msgr2=0x7f2044074010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f2044071bd0 0x7f2044074010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b64000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b64000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2b54006290 0x7f2b54052610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b64000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5404fdf0 con 0x7f2b540e33d0 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b64000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2b54052b50 con 0x7f2b54006290 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b64000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2b540500d0 con 0x7f2b54050e10 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2044068450 msgr2=0x7f204406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2044068450 0x7f204406a910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f205c009fd0 tx=0x7f205c00b040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 msgr2=0x7f20640c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 --2- 172.21.15.90:0/1180812375 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20640cc740 0x7f20640c9550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20500ab000 tx=0x7f2050019230 comp rx=0 tx=0).stop 2024-03-31T15:54:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50018/0 (socket says 172.21.15.90:50018) 2024-03-31T15:54:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 -- 172.21.15.90:0/785122134 learned_addr learned my addr 172.21.15.90:0/785122134 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2062800640 1 -- 172.21.15.90:0/1180812375 reap_dead start 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7feac7e00640 1 -- 172.21.15.90:0/3892281990 <== osd.3 v2:172.21.15.90:6808/4045683431 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feaa8002de0 con 0x7feab0071b00 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.866+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb334018020 con 0x7fb3480d2de0 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb33f400640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb328068380 0x7fb32806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb334086ad0 con 0x7fb3480d2de0 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb33f400640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7fb328071a80 0x7fb328073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb334020ec0 con 0x7fb328071a80 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb33408b2a0 con 0x7fb3480d2de0 2024-03-31T15:54:14.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7fb347400640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7fb328071a80 0x7fb328073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 shutdown_connections 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 >> 172.21.15.90:0/1180812375 conn(0x7f20640b1880 msgr2=0x7f206415d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2d0c0c7070 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2cf8068450 0x7f2cf806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2d0c128e50 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7f2cf8071b50 0x7f2cf8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d0c0adc40 con 0x7f2cf8071b50 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d1b400640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2cf8068450 0x7f2cf806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2d0c0f4c50 con 0x7f2d1c09a2c0 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 shutdown_connections 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2068a00640 1 -- 172.21.15.90:0/1180812375 wait complete. 2024-03-31T15:54:14.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 --> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fea54002de0 con 0x7fea5c071a80 2024-03-31T15:54:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2d1b400640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2cf8068450 0x7f2cf806a910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d04001720 tx=0x7f2d04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 -- 172.21.15.90:0/785122134 >> v2:172.21.15.90:3300/0 conn(0x7f2b54006290 msgr2=0x7f2b54052610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 --2- 172.21.15.90:0/785122134 >> v2:172.21.15.90:3300/0 conn(0x7f2b54006290 0x7f2b54052610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 -- 172.21.15.90:0/785122134 >> v1:172.21.15.90:6789/0 conn(0x7f2b540e33d0 legacy=0x7f2b54050700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:14.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.870+0000 7f2b60c00640 1 -- 172.21.15.90:0/785122134 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b54038ec0 con 0x7f2b54050e10 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fb346a00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb328068380 0x7fb32806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7feab0071b00 msgr2=0x7feab0073f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7feab0071b00 0x7feab0073f40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feab0068380 msgr2=0x7feab006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feab0068380 0x7feab006a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feac0000f70 tx=0x7feac0007e90 comp rx=0 tx=0).stop 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 msgr2=0x7fead0147930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 --2- 172.21.15.90:0/3892281990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead00cc740 0x7fead0147930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feac80194d0 tx=0x7feac8019660 comp rx=0 tx=0).stop 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead4a00640 1 -- 172.21.15.90:0/3892281990 reap_dead start 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 shutdown_connections 2024-03-31T15:54:14.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 >> 172.21.15.90:0/3892281990 conn(0x7fead00b1880 msgr2=0x7fead00b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2d1be00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7f2cf8071b50 0x7f2cf8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fb347400640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7fb328071a80 0x7fb328073ec0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fb346a00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb328068380 0x7fb32806a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb338006d60 tx=0x7fb338006cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea72a00640 1 -- 172.21.15.90:0/3221324341 <== osd.2 v2:172.21.15.90:6800/347246527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fea54002de0 con 0x7fea5c071a80 2024-03-31T15:54:14.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 shutdown_connections 2024-03-31T15:54:14.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fead6a00640 1 -- 172.21.15.90:0/3892281990 wait complete. 2024-03-31T15:54:14.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7fea5c071a80 msgr2=0x7fea5c073ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7fea5c071a80 0x7fea5c073ec0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2d1be00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7f2cf8071b50 0x7f2cf8073f90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== osd.7 v2:172.21.15.102:6809/552876268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb334020ec0 con 0x7fb328071a80 2024-03-31T15:54:14.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fea5c068380 msgr2=0x7fea5c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fea5c068380 0x7fea5c06a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fea6c005890 tx=0x7fea6c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 msgr2=0x7fea7c162f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 --2- 172.21.15.90:0/3221324341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c072c30 0x7fea7c162f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fea68000f70 tx=0x7fea6800c890 comp rx=0 tx=0).stop 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea7b400640 1 -- 172.21.15.90:0/3221324341 reap_dead start 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 shutdown_connections 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 >> 172.21.15.90:0/3221324341 conn(0x7fea7c0b1a50 msgr2=0x7fea7c0cc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 shutdown_connections 2024-03-31T15:54:14.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2b60c00640 1 --2- 172.21.15.90:0/785122134 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009870 tx=0x7f2b4801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=218536a65b7963aa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7fea81800640 1 -- 172.21.15.90:0/3221324341 wait complete. 2024-03-31T15:54:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.874+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== osd.5 v2:172.21.15.102:6808/3785791459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d0c0adc40 con 0x7f2cf8071b50 2024-03-31T15:54:14.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b5be00640 1 -- 172.21.15.90:0/785122134 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b480aa650 con 0x7f2b54050e10 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b5be00640 1 -- 172.21.15.90:0/785122134 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b480af070 con 0x7f2b54050e10 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b5be00640 1 -- 172.21.15.90:0/785122134 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b480aaa10 con 0x7f2b54050e10 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 -- 172.21.15.90:0/785122134 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 msgr2=0x7f2b5403a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 --2- 172.21.15.90:0/785122134 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009870 tx=0x7f2b4801c510 comp rx=0 tx=0).stop 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 -- 172.21.15.90:0/785122134 shutdown_connections 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 --2- 172.21.15.90:0/785122134 >> v2:172.21.15.90:3300/0 conn(0x7f2b54006290 0x7f2b54052610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 --2- 172.21.15.90:0/785122134 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b54050e10 0x7f2b5403a640 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b48009870 tx=0x7f2b4801c510 comp rx=0 tx=0).stop 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.878+0000 7f2b64000640 1 -- 172.21.15.90:0/785122134 >> 172.21.15.90:0/785122134 conn(0x7f2b54009fc0 msgr2=0x7f2b5400a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.882+0000 7f2b64000640 1 -- 172.21.15.90:0/785122134 shutdown_connections 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.882+0000 7f2b64000640 1 -- 172.21.15.90:0/785122134 wait complete. 2024-03-31T15:54:14.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.882+0000 7f2b64000640 1 Processor -- start 2024-03-31T15:54:14.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b64000640 1 -- start start 2024-03-31T15:54:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b64000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 0x7f2b540d3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b64000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b540e33d0 0x7f2b54048ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b64000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b5404ae10 0x7f2b540d3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b60c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b540e33d0 0x7f2b54048ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.890+0000 7f2b60c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b540e33d0 0x7f2b54048ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47046/0 (socket says 172.21.15.90:47046) 2024-03-31T15:54:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 0x7f2b540d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 0x7f2b540d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51374/0 (socket says 172.21.15.90:51374) 2024-03-31T15:54:14.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b60c00640 1 -- 172.21.15.90:0/1576864593 learned_addr learned my addr 172.21.15.90:0/1576864593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b64000640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2b54039ed0 con 0x7f2b54006290 2024-03-31T15:54:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b64000640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2b540e37b0 con 0x7f2b540e33d0 2024-03-31T15:54:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b64000640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2b5403a050 con 0x7f2b5404ae10 2024-03-31T15:54:14.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b62000640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b5404ae10 0x7f2b540d3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 -- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b5404ae10 msgr2=0x7f2b540d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2b5404ae10 0x7f2b540d3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 -- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b540e33d0 msgr2=0x7f2b54048ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b540e33d0 0x7f2b54048ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b54048ff0 con 0x7f2b54006290 2024-03-31T15:54:14.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b61600640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 0x7f2b540d3110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000b410 tx=0x7f2b5000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50015020 con 0x7f2b54006290 2024-03-31T15:54:14.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.894+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b50011830 con 0x7f2b54006290 2024-03-31T15:54:14.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.898+0000 7f2b64000640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b540cfe30 con 0x7f2b54006290 2024-03-31T15:54:14.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.898+0000 7f2b64000640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b540cfb50 con 0x7f2b54006290 2024-03-31T15:54:14.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.898+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50011c20 con 0x7f2b54006290 2024-03-31T15:54:14.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.898+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 --> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb320002de0 con 0x7fb328071a80 2024-03-31T15:54:14.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b540053a0 con 0x7f2b54006290 2024-03-31T15:54:14.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2b50018020 con 0x7f2b54006290 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb33f400640 1 -- 172.21.15.90:0/1164036685 <== osd.7 v2:172.21.15.102:6809/552876268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb320002de0 con 0x7fb328071a80 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7fb328071a80 msgr2=0x7fb328073ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7fb328071a80 0x7fb328073ec0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb328068380 msgr2=0x7fb32806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb328068380 0x7fb32806a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb338006d60 tx=0x7fb338006cb0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 msgr2=0x7fb348159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 --2- 172.21.15.90:0/1164036685 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3480d2de0 0x7fb348159120 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb33400b410 tx=0x7fb33400b8d0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb347400640 1 -- 172.21.15.90:0/1164036685 reap_dead start 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 shutdown_connections 2024-03-31T15:54:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 >> 172.21.15.90:0/1164036685 conn(0x7fb3480d40f0 msgr2=0x7fb3481587d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2b3c068450 0x7f2b3c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.902+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 --> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d1c09b2a0 con 0x7f2cf8071b50 2024-03-31T15:54:14.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2b500869f0 con 0x7f2b54006290 2024-03-31T15:54:14.901 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f2b3c071bc0 0x7f2b3c074000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b5001deb0 con 0x7f2b3c071bc0 2024-03-31T15:54:14.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-31T15:54:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2b5008b2a0 con 0x7f2b54006290 2024-03-31T15:54:14.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d13400640 1 -- 172.21.15.90:0/489706889 <== osd.5 v2:172.21.15.102:6808/3785791459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2d1c09b2a0 con 0x7f2cf8071b50 2024-03-31T15:54:14.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b62000640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f2b3c071bc0 0x7f2b3c074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b60c00640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2b3c068450 0x7f2b3c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:14.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 shutdown_connections 2024-03-31T15:54:14.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7fb34de00640 1 -- 172.21.15.90:0/1164036685 wait complete. 2024-03-31T15:54:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b62000640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f2b3c071bc0 0x7f2b3c074000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7f2cf8071b50 msgr2=0x7f2cf8073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7f2cf8071b50 0x7f2cf8073f90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b60c00640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2b3c068450 0x7f2b3c06a910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b4801a140 tx=0x7f2b480a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2cf8068450 msgr2=0x7f2cf806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2cf8068450 0x7f2cf806a910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2d04001720 tx=0x7f2d04009040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 msgr2=0x7f2d1c0c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 --2- 172.21.15.90:0/489706889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d1c09a2c0 0x7f2d1c0c5f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0b8040 tx=0x7f2d0c022040 comp rx=0 tx=0).stop 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d1be00640 1 -- 172.21.15.90:0/489706889 reap_dead start 2024-03-31T15:54:14.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 shutdown_connections 2024-03-31T15:54:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 >> 172.21.15.90:0/489706889 conn(0x7f2d1c0d03e0 msgr2=0x7f2d1c0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== osd.6 v2:172.21.15.102:6824/948959501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b5001deb0 con 0x7f2b3c071bc0 2024-03-31T15:54:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 shutdown_connections 2024-03-31T15:54:14.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.906+0000 7f2d10c00640 1 -- 172.21.15.90:0/489706889 wait complete. 2024-03-31T15:54:14.910 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:14.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-31T15:54:14.916 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-03-31T15:54:14.917 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-03-31T15:54:14.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-31T15:54:14.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-31T15:54:14.919 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:14.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-31T15:54:14.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.930+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 --> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b54054030 con 0x7f2b3c071bc0 2024-03-31T15:54:14.930 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:14.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b59600640 1 -- 172.21.15.90:0/1576864593 <== osd.6 v2:172.21.15.102:6824/948959501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2b54054030 con 0x7f2b3c071bc0 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f2b3c071bc0 msgr2=0x7f2b3c074000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f2b3c071bc0 0x7f2b3c074000 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2b3c068450 msgr2=0x7f2b3c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2b3c068450 0x7f2b3c06a910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b4801a140 tx=0x7f2b480a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 msgr2=0x7f2b540d3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 --2- 172.21.15.90:0/1576864593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2b54006290 0x7f2b540d3110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000b410 tx=0x7f2b5000b810 comp rx=0 tx=0).stop 2024-03-31T15:54:14.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b62000640 1 -- 172.21.15.90:0/1576864593 reap_dead start 2024-03-31T15:54:14.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-31T15:54:14.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 shutdown_connections 2024-03-31T15:54:14.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 >> 172.21.15.90:0/1576864593 conn(0x7f2b54009fc0 msgr2=0x7f2b540de450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:14.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 shutdown_connections 2024-03-31T15:54:14.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:14.934+0000 7f2b4ea00640 1 -- 172.21.15.90:0/1576864593 wait complete. 2024-03-31T15:54:14.939 INFO:teuthology.orchestra.run.smithi090.stdout:55834574858 2024-03-31T15:54:14.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-31T15:54:14.965 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:14.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-31T15:54:15.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.154+0000 7fc613200640 1 Processor -- start 2024-03-31T15:54:15.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 -- start start 2024-03-31T15:54:15.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc60c0529b0 0x7fc60c055d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc60c034850 0x7fc60c034c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60c007f20 con 0x7fc60c0df930 2024-03-31T15:54:15.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc60c0081c0 con 0x7fc60c0529b0 2024-03-31T15:54:15.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc613200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc60c03a250 con 0x7fc60c034850 2024-03-31T15:54:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc612800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc60c034850 0x7fc60c034c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc612800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc60c034850 0x7fc60c034c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47060/0 (socket says 172.21.15.90:47060) 2024-03-31T15:54:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc612800640 1 -- 172.21.15.90:0/1012418274 learned_addr learned my addr 172.21.15.90:0/1012418274 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc612800640 1 -- 172.21.15.90:0/1012418274 >> v2:172.21.15.90:3300/0 conn(0x7fc60c0529b0 msgr2=0x7fc60c055d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:54:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.158+0000 7fc612800640 1 --2- 172.21.15.90:0/1012418274 >> v2:172.21.15.90:3300/0 conn(0x7fc60c0529b0 0x7fc60c055d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.162+0000 7fc612800640 1 -- 172.21.15.90:0/1012418274 >> v1:172.21.15.90:6789/0 conn(0x7fc60c0df930 legacy=0x7fc60c03aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.162+0000 7fc610a00640 1 -- 172.21.15.90:0/1012418274 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124928068 0 0) 0x7fc60c007f20 con 0x7fc60c0df930 2024-03-31T15:54:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.162+0000 7fc612800640 1 -- 172.21.15.90:0/1012418274 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c03a530 con 0x7fc60c034850 2024-03-31T15:54:15.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc612800640 1 --2- 172.21.15.90:0/1012418274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c034c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc60c03b410 tx=0x7fc600020120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7c097fe8b9ff2b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7f5409a00640 1 Processor -- start 2024-03-31T15:54:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7f5409a00640 1 -- start start 2024-03-31T15:54:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc610a00640 1 -- 172.21.15.90:0/1012418274 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000b5070 con 0x7fc60c034850 2024-03-31T15:54:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc610a00640 1 -- 172.21.15.90:0/1012418274 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6000b0600 con 0x7fc60c034850 2024-03-31T15:54:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc610a00640 1 -- 172.21.15.90:0/1012418274 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60000b040 con 0x7fc60c034850 2024-03-31T15:54:15.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc613200640 1 -- 172.21.15.90:0/1012418274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 msgr2=0x7fc60c034c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.166+0000 7fc613200640 1 --2- 172.21.15.90:0/1012418274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c034c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc60c03b410 tx=0x7fc600020120 comp rx=0 tx=0).stop 2024-03-31T15:54:15.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f540415a100 0x7f54040710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f54040ca120 0x7f54040ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540415a4e0 con 0x7f54040715e0 2024-03-31T15:54:15.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f54040cc440 con 0x7f54040ca120 2024-03-31T15:54:15.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f54040cdfa0 con 0x7f540415a100 2024-03-31T15:54:15.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f54040ca120 0x7f54040ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f54040ca120 0x7f54040ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51382/0 (socket says 172.21.15.90:51382) 2024-03-31T15:54:15.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 -- 172.21.15.90:0/3113515785 learned_addr learned my addr 172.21.15.90:0/3113515785 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 -- 172.21.15.90:0/3113515785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f540415a100 msgr2=0x7f54040710a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:54:15.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 --2- 172.21.15.90:0/3113515785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f540415a100 0x7f54040710a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 -- 172.21.15.90:0/3113515785 >> v1:172.21.15.90:6789/0 conn(0x7f54040715e0 legacy=0x7f54040ce430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.170+0000 7f5409000640 1 -- 172.21.15.90:0/3113515785 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540415e870 con 0x7f54040ca120 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 -- 172.21.15.90:0/1012418274 shutdown_connections 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 --2- 172.21.15.90:0/1012418274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c034c80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 --2- 172.21.15.90:0/1012418274 >> v2:172.21.15.90:3300/0 conn(0x7fc60c0529b0 0x7fc60c055d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 -- 172.21.15.90:0/1012418274 >> 172.21.15.90:0/1012418274 conn(0x7fc60c01e6d0 msgr2=0x7fc60c055400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 -- 172.21.15.90:0/1012418274 shutdown_connections 2024-03-31T15:54:15.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 -- 172.21.15.90:0/1012418274 wait complete. 2024-03-31T15:54:15.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 Processor -- start 2024-03-31T15:54:15.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.174+0000 7fc613200640 1 -- start start 2024-03-31T15:54:15.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 Processor -- start 2024-03-31T15:54:15.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 -- start start 2024-03-31T15:54:15.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5409000640 1 --2- 172.21.15.90:0/3113515785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040ca120 0x7f54040ca5b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009ad0 tx=0x7f53f001c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75dfabb907890fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c052fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 0x7fc60c054290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5402a00640 1 -- 172.21.15.90:0/3113515785 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f00a9400 con 0x7f54040ca120 2024-03-31T15:54:15.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5402a00640 1 -- 172.21.15.90:0/3113515785 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f00a95a0 con 0x7f54040ca120 2024-03-31T15:54:15.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5402a00640 1 -- 172.21.15.90:0/3113515785 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f00a9970 con 0x7f54040ca120 2024-03-31T15:54:15.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc611400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 0x7fc60c054290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc611400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 0x7fc60c054290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50040/0 (socket says 172.21.15.90:50040) 2024-03-31T15:54:15.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51398/0 (socket says 172.21.15.90:51398) 2024-03-31T15:54:15.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5409a00640 1 -- 172.21.15.90:0/3113515785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040ca120 msgr2=0x7f54040ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7f5409a00640 1 --2- 172.21.15.90:0/3113515785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040ca120 0x7f54040ca5b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009ad0 tx=0x7f53f001c7a0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff54c071500 0x7ff54c0ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54c15eb50 con 0x7ff54c0710a0 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff54c0cc440 con 0x7ff54c071500 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54be00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff54c0cdfa0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54aa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50050/0 (socket says 172.21.15.90:50050) 2024-03-31T15:54:15.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 -- 172.21.15.90:0/1473794848 learned_addr learned my addr 172.21.15.90:0/1473794848 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc611400640 1 -- 172.21.15.90:0/1887728571 learned_addr learned my addr 172.21.15.90:0/1887728571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c0062d0 con 0x7fc60c0df930 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c056260 con 0x7fc60c034850 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c006450 con 0x7fc60c0529b0 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 -- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 msgr2=0x7fc60c054290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 0x7fc60c054290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 -- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 msgr2=0x7fc60c052fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c052fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0538b0 con 0x7fc60c0df930 2024-03-31T15:54:15.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc612800640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6000b0000 tx=0x7fc60001e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000b5040 con 0x7fc60c0df930 2024-03-31T15:54:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6000c8070 con 0x7fc60c0df930 2024-03-31T15:54:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6000ba810 con 0x7fc60c0df930 2024-03-31T15:54:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.178+0000 7ff54a000640 1 --1- 172.21.15.90:0/1473794848 >> v1:172.21.15.102:6789/0 conn(0x7ff54c0710a0 0x7ff54c0ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45432/0 (socket says 172.21.15.90:45432) 2024-03-31T15:54:15.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 Processor -- start 2024-03-31T15:54:15.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54b400640 1 --2- 172.21.15.90:0/1473794848 >> v2:172.21.15.90:3300/0 conn(0x7ff54c071500 0x7ff54c0ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 -- start start 2024-03-31T15:54:15.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 -- 172.21.15.90:0/1473794848 >> v2:172.21.15.90:3300/0 conn(0x7ff54c071500 msgr2=0x7ff54c0ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 --2- 172.21.15.90:0/1473794848 >> v2:172.21.15.90:3300/0 conn(0x7ff54c071500 0x7ff54c0ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 -- 172.21.15.90:0/1473794848 >> v1:172.21.15.102:6789/0 conn(0x7ff54c0710a0 legacy=0x7ff54c0ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7ff54aa00640 1 -- 172.21.15.90:0/1473794848 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c0cd950 con 0x7ff54c0b21c0 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 -- 172.21.15.90:0/3113515785 shutdown_connections 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 --2- 172.21.15.90:0/3113515785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040ca120 0x7f54040ca5b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 --2- 172.21.15.90:0/3113515785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f540415a100 0x7f54040710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 -- 172.21.15.90:0/3113515785 >> 172.21.15.90:0/3113515785 conn(0x7f54040b16a0 msgr2=0x7f54040b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c1782c0 con 0x7fc60c0df930 2024-03-31T15:54:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c178880 con 0x7fc60c0df930 2024-03-31T15:54:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 -- 172.21.15.90:0/3113515785 shutdown_connections 2024-03-31T15:54:15.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7f5409a00640 1 -- 172.21.15.90:0/3113515785 wait complete. 2024-03-31T15:54:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb990144880 0x7fb9900d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb9900d0fd0 0x7fb9900721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb98f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb987e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb990144880 0x7fb9900d0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb987e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb990144880 0x7fb9900d0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47086/0 (socket says 172.21.15.90:47086) 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb98f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb987e00640 1 -- 172.21.15.90:0/3684777717 learned_addr learned my addr 172.21.15.90:0/3684777717 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.182+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb9900d1440 con 0x7fb9900d0fd0 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb990144150 con 0x7fb990144880 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb990161d50 con 0x7fb990144330 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 Processor -- start 2024-03-31T15:54:15.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 -- start start 2024-03-31T15:54:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c0053a0 con 0x7fc60c0df930 2024-03-31T15:54:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb987e00640 1 -- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 msgr2=0x7fb990143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb987e00640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb987e00640 1 -- 172.21.15.90:0/3684777717 >> v2:172.21.15.90:3300/0 conn(0x7fb9900d0fd0 msgr2=0x7fb9900721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb987e00640 1 --2- 172.21.15.90:0/3684777717 >> v2:172.21.15.90:3300/0 conn(0x7fb9900d0fd0 0x7fb9900721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb987e00640 1 -- 172.21.15.90:0/3684777717 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99015ed30 con 0x7fb990144880 2024-03-31T15:54:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7fb98f400640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:15.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040715e0 0x7f54041fb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 0x7f54041fd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 0x7f54040c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f54040cecc0 con 0x7f54040715e0 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f54040ceb40 con 0x7f540415a100 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5404169350 con 0x7f54040ca120 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5403e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040715e0 0x7f54041fb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5403400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 0x7f54041fd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5403400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 0x7f54041fd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50074/0 (socket says 172.21.15.90:50074) 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5403400640 1 -- 172.21.15.90:0/1178694764 learned_addr learned my addr 172.21.15.90:0/1178694764 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 0x7f54040c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 -- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 msgr2=0x7f54041fd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 0x7f54041fd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 -- 172.21.15.90:0/1178694764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040715e0 msgr2=0x7f54041fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040715e0 0x7f54041fb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.186+0000 7f5409000640 1 -- 172.21.15.90:0/1178694764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5404071aa0 con 0x7f540415a100 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f5409000640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 0x7f54040c93e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f53f00aa000 tx=0x7f53f001afe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc6000ba9b0 con 0x7fc60c0df930 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f00bc650 con 0x7f540415a100 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f00bc810 con 0x7f540415a100 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f00bcc00 con 0x7f540415a100 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5404071d80 con 0x7f540415a100 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5404072340 con 0x7f540415a100 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7ff54aa00640 1 --2- 172.21.15.90:0/1473794848 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009870 tx=0x7ff53c01ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca0b0df80f5b516c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7ff549600640 1 -- 172.21.15.90:0/1473794848 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c0b5070 con 0x7ff54c0b21c0 2024-03-31T15:54:15.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7ff549600640 1 -- 172.21.15.90:0/1473794848 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c0a89d0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7ff549600640 1 -- 172.21.15.90:0/1473794848 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c0a8da0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7fc60a000640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 0x7fc5f406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7fb987e00640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb990144880 0x7fb9900d0a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb99006d660 tx=0x7fb97801c260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84657589307b67f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7f53f9600640 1 -- 172.21.15.90:0/1178694764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54040685f0 con 0x7f540415a100 2024-03-31T15:54:15.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc60012dbf0 con 0x7fc60c0df930 2024-03-31T15:54:15.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.190+0000 7fc611e00640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 0x7fc5f406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fc611e00640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 0x7fc5f406a890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc604000f70 tx=0x7fc6040079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb98ea00640 1 -- 172.21.15.90:0/3684777717 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9780b0070 con 0x7fb990144880 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb98ea00640 1 -- 172.21.15.90:0/3684777717 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9780ab600 con 0x7fb990144880 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb98ea00640 1 -- 172.21.15.90:0/3684777717 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9780ab9d0 con 0x7fb990144880 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb990144880 msgr2=0x7fb9900d0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb990144880 0x7fb9900d0a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb99006d660 tx=0x7fb97801c260 comp rx=0 tx=0).stop 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 -- 172.21.15.90:0/1473794848 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 msgr2=0x7ff54c159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 --2- 172.21.15.90:0/1473794848 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009870 tx=0x7ff53c01ed10 comp rx=0 tx=0).stop 2024-03-31T15:54:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53f00c7430 con 0x7f540415a100 2024-03-31T15:54:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 shutdown_connections 2024-03-31T15:54:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 --2- 172.21.15.90:0/3684777717 >> v2:172.21.15.90:3300/0 conn(0x7fb9900d0fd0 0x7fb9900721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb990144880 0x7fb9900d0a90 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb99006d660 tx=0x7fb97801c260 comp rx=0 tx=0).stop 2024-03-31T15:54:15.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 --2- 172.21.15.90:0/3684777717 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb990143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 >> 172.21.15.90:0/3684777717 conn(0x7fb99006c700 msgr2=0x7fb99006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 -- 172.21.15.90:0/1473794848 shutdown_connections 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 --2- 172.21.15.90:0/1473794848 >> v2:172.21.15.90:3300/0 conn(0x7ff54c071500 0x7ff54c0ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 --2- 172.21.15.90:0/1473794848 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c159ea0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 -- 172.21.15.90:0/1473794848 >> 172.21.15.90:0/1473794848 conn(0x7ff54c0b16a0 msgr2=0x7ff54c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 -- 172.21.15.90:0/1473794848 shutdown_connections 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7ff54be00640 1 -- 172.21.15.90:0/1473794848 wait complete. 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7f53fbe00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 0x7f53e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f53f0121290 con 0x7f540415a100 2024-03-31T15:54:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 shutdown_connections 2024-03-31T15:54:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7fb995200640 1 -- 172.21.15.90:0/3684777717 wait complete. 2024-03-31T15:54:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.194+0000 7f5403e00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 0x7f53e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7f5403e00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 0x7f53e406a840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00b450 tx=0x7f53fc00d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54be00640 1 Processor -- start 2024-03-31T15:54:15.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54be00640 1 -- start start 2024-03-31T15:54:15.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc600019c80 con 0x7fc60c0df930 2024-03-31T15:54:15.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7fb995200640 1 Processor -- start 2024-03-31T15:54:15.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7fb995200640 1 -- start start 2024-03-31T15:54:15.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 0x7ff54c0c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 0x7ff54c14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 0x7ff54c14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 0x7ff54c14ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51442/0 (socket says 172.21.15.90:51442) 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 0x7ff54c0c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 0x7ff54c0c20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47112/0 (socket says 172.21.15.90:47112) 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.198+0000 7ff54b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50086/0 (socket says 172.21.15.90:50086) 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54a000640 1 -- 172.21.15.90:0/990439610 learned_addr learned my addr 172.21.15.90:0/990439610 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c15a560 con 0x7ff54c071500 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff54c15a3e0 con 0x7ff54c0710a0 2024-03-31T15:54:15.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f53f00c3030 con 0x7f540415a100 2024-03-31T15:54:15.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff54c0ceb40 con 0x7ff54c0b21c0 2024-03-31T15:54:15.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb99015c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 -- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 msgr2=0x7ff54c0c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 0x7ff54c0c20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 -- 172.21.15.90:0/990439610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 msgr2=0x7ff54c14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 0x7ff54c14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54c15cb50 con 0x7ff54c0b21c0 2024-03-31T15:54:15.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb987e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb99015c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb987e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb99015c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50098/0 (socket says 172.21.15.90:50098) 2024-03-31T15:54:15.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47118/0 (socket says 172.21.15.90:47118) 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51452/0 (socket says 172.21.15.90:51452) 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb987e00640 1 -- 172.21.15.90:0/3874163583 learned_addr learned my addr 172.21.15.90:0/3874163583 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb99006b5f0 con 0x7fb990142760 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb99015e9e0 con 0x7fb9900d0fd0 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb99006b770 con 0x7fb990144330 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 -- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 msgr2=0x7fb99015c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb99015c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 -- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 msgr2=0x7fb990142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7fb98fe00640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99015d1c0 con 0x7fb990142760 2024-03-31T15:54:15.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.202+0000 7ff54b400640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff54400ccb0 tx=0x7ff54400a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb98fe00640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb98800be20 tx=0x7fb98800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544003ba0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff544003d40 con 0x7ff54c0b21c0 2024-03-31T15:54:15.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff544004240 con 0x7ff54c0b21c0 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb988019070 con 0x7fb990142760 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb988014440 con 0x7fb990142760 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb988014830 con 0x7fb990142760 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99015d4a0 con 0x7fb990142760 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.206+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99015da60 con 0x7fb990142760 2024-03-31T15:54:15.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.210+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54c15ce10 con 0x7ff54c0b21c0 2024-03-31T15:54:15.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.210+0000 7fb98f400640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:15.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.210+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9900685f0 con 0x7fb990142760 2024-03-31T15:54:15.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.210+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54c15d3d0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5440043e0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54c0685f0 con 0x7ff54c0b21c0 2024-03-31T15:54:15.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7ff542a00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 0x7ff52c06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff544087630 con 0x7ff54c0b21c0 2024-03-31T15:54:15.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb98801c070 con 0x7fb990142760 2024-03-31T15:54:15.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.214+0000 7fb987400640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 0x7fb96c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.218+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb988087080 con 0x7fb990142760 2024-03-31T15:54:15.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.218+0000 7ff54aa00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 0x7ff52c06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.218+0000 7fb98f400640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 0x7fb96c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.218+0000 7fb98f400640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 0x7fb96c06a910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb9900ced50 tx=0x7fb980008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.222+0000 7ff54aa00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 0x7ff52c06a940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b2000 tx=0x7ff53c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.226+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff544057920 con 0x7ff54c0b21c0 2024-03-31T15:54:15.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.226+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb988023020 con 0x7fb990142760 2024-03-31T15:54:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.230+0000 7ff07fe00640 1 Processor -- start 2024-03-31T15:54:15.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.230+0000 7ff07fe00640 1 -- start start 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 0x7ff080159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff080071500 0x7ff0800ca120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff08015eb50 con 0x7ff0800710a0 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff0800cc440 con 0x7ff080071500 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff0800cdfa0 con 0x7ff0800b21c0 2024-03-31T15:54:15.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07e000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0800710a0 0x7ff0800ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33918/0 (socket says 172.21.15.90:33918) 2024-03-31T15:54:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07e000640 1 -- 172.21.15.90:0/1809756344 learned_addr learned my addr 172.21.15.90:0/1809756344 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07f400640 1 --2- 172.21.15.90:0/1809756344 >> v2:172.21.15.102:3300/0 conn(0x7ff080071500 0x7ff0800ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07ea00640 1 --2- 172.21.15.90:0/1809756344 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 0x7ff080159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07d600640 1 -- 172.21.15.90:0/1809756344 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4159206329 0 0) 0x7ff08015eb50 con 0x7ff0800710a0 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07d600640 1 -- 172.21.15.90:0/1809756344 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff064003530 con 0x7ff0800710a0 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07ea00640 1 -- 172.21.15.90:0/1809756344 >> v2:172.21.15.102:3300/0 conn(0x7ff080071500 msgr2=0x7ff0800ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07ea00640 1 --2- 172.21.15.90:0/1809756344 >> v2:172.21.15.102:3300/0 conn(0x7ff080071500 0x7ff0800ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07ea00640 1 -- 172.21.15.90:0/1809756344 >> v1:172.21.15.90:6789/0 conn(0x7ff0800710a0 legacy=0x7ff0800ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.238+0000 7ff07ea00640 1 -- 172.21.15.90:0/1809756344 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0800cd950 con 0x7ff0800b21c0 2024-03-31T15:54:15.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.242+0000 7ff07ea00640 1 --2- 172.21.15.90:0/1809756344 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 0x7ff080159ea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097c0 tx=0x7ff06c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726598250e3aa748 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.242+0000 7ff3be200640 1 Processor -- start 2024-03-31T15:54:15.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07d600640 1 -- 172.21.15.90:0/1809756344 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0b5070 con 0x7ff0800b21c0 2024-03-31T15:54:15.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.242+0000 7ff3be200640 1 -- start start 2024-03-31T15:54:15.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07d600640 1 -- 172.21.15.90:0/1809756344 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06c0a89d0 con 0x7ff0800b21c0 2024-03-31T15:54:15.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07d600640 1 -- 172.21.15.90:0/1809756344 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0b5070 con 0x7ff0800b21c0 2024-03-31T15:54:15.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 -- 172.21.15.90:0/1809756344 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 msgr2=0x7ff080159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 --2- 172.21.15.90:0/1809756344 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 0x7ff080159ea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097c0 tx=0x7ff06c01ecd0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 -- 172.21.15.90:0/1809756344 shutdown_connections 2024-03-31T15:54:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3be200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 0x7ff3b80d2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3be200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff3b80cf270 0x7ff3b80c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3be200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b80cd4e0 con 0x7ff3b80ce780 2024-03-31T15:54:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3be200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b80cd7c0 con 0x7ff3b80cf270 2024-03-31T15:54:15.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3be200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b80cdaa0 con 0x7ff3b815c350 2024-03-31T15:54:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 --2- 172.21.15.90:0/1809756344 >> v2:172.21.15.102:3300/0 conn(0x7ff080071500 0x7ff0800ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 --2- 172.21.15.90:0/1809756344 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800b21c0 0x7ff080159ea0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097c0 tx=0x7ff06c01ecd0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 -- 172.21.15.90:0/1809756344 >> 172.21.15.90:0/1809756344 conn(0x7ff0800b16a0 msgr2=0x7ff0800b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 -- 172.21.15.90:0/1809756344 shutdown_connections 2024-03-31T15:54:15.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3b7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff3b80cf270 0x7ff3b80c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff07fe00640 1 -- 172.21.15.90:0/1809756344 wait complete. 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3b6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 0x7ff3b80d2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3b6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 0x7ff3b80d2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47144/0 (socket says 172.21.15.90:47144) 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3b6a00640 1 -- 172.21.15.90:0/609034046 learned_addr learned my addr 172.21.15.90:0/609034046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.246+0000 7ff3b7e00640 1 -- 172.21.15.90:0/609034046 >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 msgr2=0x7ff3b80d2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff3b7e00640 1 --2- 172.21.15.90:0/609034046 >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 0x7ff3b80d2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff3b7e00640 1 -- 172.21.15.90:0/609034046 >> v1:172.21.15.102:6789/0 conn(0x7ff3b80ce780 legacy=0x7ff3b80ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff3b7e00640 1 -- 172.21.15.90:0/609034046 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b80c8f90 con 0x7ff3b80cf270 2024-03-31T15:54:15.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff07fe00640 1 Processor -- start 2024-03-31T15:54:15.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff07fe00640 1 -- start start 2024-03-31T15:54:15.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 0x7ff08016bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.250+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0800c2120 0x7ff0800a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff95b800640 1 Processor -- start 2024-03-31T15:54:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 0x7ff08016bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47152/0 (socket says 172.21.15.90:47152) 2024-03-31T15:54:15.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 0x7ff08016bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50104/0 (socket says 172.21.15.90:50104) 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 -- 172.21.15.90:0/2439132162 learned_addr learned my addr 172.21.15.90:0/2439132162 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff08015a560 con 0x7ff0800c2120 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff08015a3e0 con 0x7ff080071500 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff0800ceb40 con 0x7ff0800710a0 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff95b800640 1 -- start start 2024-03-31T15:54:15.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 -- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 msgr2=0x7ff08016bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 0x7ff08016bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 -- 172.21.15.90:0/2439132162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0800c2120 msgr2=0x7ff0800a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0800c2120 0x7ff0800a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08006de50 con 0x7ff080071500 2024-03-31T15:54:15.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07e000640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c450 tx=0x7ff07000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff3b7e00640 1 --2- 172.21.15.90:0/609034046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80cf270 0x7ff3b80c94b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8067d40 tx=0x7ff3ac01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11994fab0a15dc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070017070 con 0x7ff080071500 2024-03-31T15:54:15.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08015a920 con 0x7ff080071500 2024-03-31T15:54:15.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff07fe00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08015aee0 con 0x7ff080071500 2024-03-31T15:54:15.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0800685f0 con 0x7ff080071500 2024-03-31T15:54:15.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff070005cf0 con 0x7ff080071500 2024-03-31T15:54:15.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070017070 con 0x7ff080071500 2024-03-31T15:54:15.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.254+0000 7ff3b6000640 1 -- 172.21.15.90:0/609034046 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac0b1070 con 0x7ff3b80cf270 2024-03-31T15:54:15.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff3b6000640 1 -- 172.21.15.90:0/609034046 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ac0ac630 con 0x7ff3b80cf270 2024-03-31T15:54:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff3b6000640 1 -- 172.21.15.90:0/609034046 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac0a8050 con 0x7ff3b80cf270 2024-03-31T15:54:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff3be200640 1 -- 172.21.15.90:0/609034046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80cf270 msgr2=0x7ff3b80c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff3be200640 1 --2- 172.21.15.90:0/609034046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80cf270 0x7ff3b80c94b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8067d40 tx=0x7ff3ac01c440 comp rx=0 tx=0).stop 2024-03-31T15:54:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff95b800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9540d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff953e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff9540b2fa0 0x7ff9540d26d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45448/0 (socket says 172.21.15.90:45448) 2024-03-31T15:54:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff953e00640 1 -- 172.21.15.90:0/677108275 learned_addr learned my addr 172.21.15.90:0/677108275 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9540d4e30 con 0x7ff9540d2de0 2024-03-31T15:54:15.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9540b0de0 con 0x7ff9540b2fa0 2024-03-31T15:54:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff9540d4be0 con 0x7ff954161cc0 2024-03-31T15:54:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff958e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9540d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3110074943 0 0) 0x7ff9540d4e30 con 0x7ff9540d2de0 2024-03-31T15:54:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.258+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff938003530 con 0x7ff9540d2de0 2024-03-31T15:54:15.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3307151906 0 0) 0x7ff938003530 con 0x7ff9540d2de0 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 msgr2=0x7ff9540d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9540d1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 >> v1:172.21.15.102:6789/0 conn(0x7ff9540b2fa0 legacy=0x7ff9540d26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9540b10c0 con 0x7ff9540d2de0 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7ff944001590 con 0x7ff9540d2de0 2024-03-31T15:54:15.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 >> v1:172.21.15.90:6789/0 conn(0x7ff9540d2de0 legacy=0x7ff9540b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9380039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff938004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff938004d50 0x7ff9380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff958e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9380039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9540d4be0 con 0x7ff938004d50 2024-03-31T15:54:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff07001a070 con 0x7ff080071500 2024-03-31T15:54:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff3be200640 1 -- 172.21.15.90:0/609034046 shutdown_connections 2024-03-31T15:54:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff3be200640 1 --2- 172.21.15.90:0/609034046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80cf270 0x7ff3b80c94b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff3be200640 1 --2- 172.21.15.90:0/609034046 >> v2:172.21.15.102:3300/0 conn(0x7ff3b815c350 0x7ff3b80d2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff076a00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 0x7ff06006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff3be200640 1 -- 172.21.15.90:0/609034046 >> 172.21.15.90:0/609034046 conn(0x7ff3b80d1e20 msgr2=0x7ff3b80d2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff938004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff07ea00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 0x7ff06006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff93800bbe0 con 0x7ff938004000 2024-03-31T15:54:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff93800bde0 con 0x7ff954161cc0 2024-03-31T15:54:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff070053ce0 con 0x7ff080071500 2024-03-31T15:54:15.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff959800640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff938004d50 0x7ff9380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 -- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 msgr2=0x7ff9380039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954161cc0 0x7ff9380039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 -- 172.21.15.90:0/677108275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff938004d50 msgr2=0x7ff9380051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff938004d50 0x7ff9380051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff953e00640 1 -- 172.21.15.90:0/677108275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93800c0a0 con 0x7ff938004000 2024-03-31T15:54:15.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.262+0000 7ff07ea00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 0x7ff06006aa40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0b2000 tx=0x7ff06c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.266+0000 7ff3be200640 1 -- 172.21.15.90:0/609034046 shutdown_connections 2024-03-31T15:54:15.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.266+0000 7ff3be200640 1 -- 172.21.15.90:0/609034046 wait complete. 2024-03-31T15:54:15.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.266+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff070055c40 con 0x7ff080071500 2024-03-31T15:54:15.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.266+0000 7ff3be200640 1 Processor -- start 2024-03-31T15:54:15.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.266+0000 7ff953e00640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff938004810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff940001850 tx=0x7ff94001c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=803381776a25a071 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940001470 con 0x7ff938004000 2024-03-31T15:54:15.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff940004ab0 con 0x7ff938004000 2024-03-31T15:54:15.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff953400640 1 -- 172.21.15.90:0/677108275 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940007070 con 0x7ff938004000 2024-03-31T15:54:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 msgr2=0x7ff938004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff95b800640 1 --2- 172.21.15.90:0/677108275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff938004810 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff940001850 tx=0x7ff94001c500 comp rx=0 tx=0).stop 2024-03-31T15:54:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff959800640 1 -- 172.21.15.90:0/677108275 reap_dead start 2024-03-31T15:54:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 shutdown_connections 2024-03-31T15:54:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 >> 172.21.15.90:0/677108275 conn(0x7ff9540d40f0 msgr2=0x7ff9540d44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.270+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 shutdown_connections 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 -- start start 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80ce780 0x7ff3b8143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b815c350 0x7ff3b815eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80ca590 con 0x7ff3b80ce780 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b80ca410 con 0x7ff3b815c350 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b80ca710 con 0x7ff3b80cf270 2024-03-31T15:54:15.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50132/0 (socket says 172.21.15.90:50132) 2024-03-31T15:54:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 -- 172.21.15.90:0/3865404785 learned_addr learned my addr 172.21.15.90:0/3865404785 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff95b800640 1 -- 172.21.15.90:0/677108275 wait complete. 2024-03-31T15:54:15.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff95b800640 1 Processor -- start 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 -- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b815c350 msgr2=0x7ff3b815eaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b815c350 0x7ff3b815eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 -- 172.21.15.90:0/3865404785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80ce780 msgr2=0x7ff3b8143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80ce780 0x7ff3b8143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 -- 172.21.15.90:0/3865404785 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b8141340 con 0x7ff3b80cf270 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3b6a00640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400e810 tx=0x7ff3a400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4017070 con 0x7ff3b80cf270 2024-03-31T15:54:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff95b800640 1 -- start start 2024-03-31T15:54:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a400c6d0 con 0x7ff3b80cf270 2024-03-31T15:54:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.274+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b81439a0 con 0x7ff3b80cf270 2024-03-31T15:54:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b81441e0 con 0x7ff3b80cf270 2024-03-31T15:54:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400f040 con 0x7ff3b80cf270 2024-03-31T15:54:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff953e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff953e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50148/0 (socket says 172.21.15.90:50148) 2024-03-31T15:54:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff958e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff958e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47170/0 (socket says 172.21.15.90:47170) 2024-03-31T15:54:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff3a401a070 con 0x7ff3b80cf270 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff959800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff959800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51488/0 (socket says 172.21.15.90:51488) 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff953e00640 1 -- 172.21.15.90:0/54722607 learned_addr learned my addr 172.21.15.90:0/54722607 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff9540d22a0 con 0x7ff95406cc50 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff9540d2120 con 0x7ff938004000 2024-03-31T15:54:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.278+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff9540b1430 con 0x7ff954150410 2024-03-31T15:54:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff3ab400640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 0x7ff39806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff3a40868a0 con 0x7ff3b80cf270 2024-03-31T15:54:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff3b7400640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 0x7ff39806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3940052a0 con 0x7ff3b80cf270 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 -- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 msgr2=0x7ff95406c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 -- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 msgr2=0x7ff9540b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9540cdd60 con 0x7ff95406cc50 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff958e00640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff959800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff93800bfb0 tx=0x7ff9440062d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff3b7400640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 0x7ff39806a8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000ab90 tx=0x7ff3b0009230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94401a070 con 0x7ff95406cc50 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9440050d0 con 0x7ff95406cc50 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9440054c0 con 0x7ff95406cc50 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9540b6e00 con 0x7ff95406cc50 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9540b73c0 con 0x7ff95406cc50 2024-03-31T15:54:15.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.282+0000 7ff953e00640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:15.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.286+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff94401d070 con 0x7ff95406cc50 2024-03-31T15:54:15.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9540b75c0 con 0x7ff95406cc50 2024-03-31T15:54:15.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff950c00640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 0x7ff93406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff9440864a0 con 0x7ff95406cc50 2024-03-31T15:54:15.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff3b81439a0 con 0x7ff3b80cf270 2024-03-31T15:54:15.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff958e00640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 0x7ff93406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.290+0000 7ff958e00640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 0x7ff93406a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009fd0 tx=0x7ff94c009760 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.298+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff9540b75c0 con 0x7ff95406cc50 2024-03-31T15:54:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.326+0000 7feb34a00640 1 Processor -- start 2024-03-31T15:54:15.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.326+0000 7feb34a00640 1 -- start start 2024-03-31T15:54:15.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb34a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb3015a100 0x7feb300710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb34a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3015a4e0 con 0x7feb300715e0 2024-03-31T15:54:15.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb34a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3015e870 con 0x7feb300b21c0 2024-03-31T15:54:15.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb34a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7feb300cc440 con 0x7feb3015a100 2024-03-31T15:54:15.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb3015a100 0x7feb300710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2f200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feb300b21c0 0x7feb3015dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45454/0 (socket says 172.21.15.90:45454) 2024-03-31T15:54:15.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2f200640 1 -- 172.21.15.90:0/3947306555 learned_addr learned my addr 172.21.15.90:0/3947306555 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:15.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2e800640 1 -- 172.21.15.90:0/3947306555 >> v1:172.21.15.102:6789/0 conn(0x7feb300b21c0 legacy=0x7feb3015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2e800640 1 -- 172.21.15.90:0/3947306555 >> v1:172.21.15.90:6789/0 conn(0x7feb300715e0 legacy=0x7feb300ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.330+0000 7feb2e800640 1 -- 172.21.15.90:0/3947306555 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3015eb50 con 0x7feb3015a100 2024-03-31T15:54:15.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb2e800640 1 --2- 172.21.15.90:0/3947306555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300710a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb28009d90 tx=0x7feb2801c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1d32393125669e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb2d400640 1 -- 172.21.15.90:0/3947306555 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb280b2070 con 0x7feb3015a100 2024-03-31T15:54:15.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb2d400640 1 -- 172.21.15.90:0/3947306555 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb280a5880 con 0x7feb3015a100 2024-03-31T15:54:15.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb2d400640 1 -- 172.21.15.90:0/3947306555 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb280a5c50 con 0x7feb3015a100 2024-03-31T15:54:15.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb34a00640 1 -- 172.21.15.90:0/3947306555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 msgr2=0x7feb300710a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.342+0000 7feb34a00640 1 --2- 172.21.15.90:0/3947306555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300710a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb28009d90 tx=0x7feb2801c820 comp rx=0 tx=0).stop 2024-03-31T15:54:15.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 -- 172.21.15.90:0/3947306555 shutdown_connections 2024-03-31T15:54:15.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 --2- 172.21.15.90:0/3947306555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300710a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 -- 172.21.15.90:0/3947306555 >> 172.21.15.90:0/3947306555 conn(0x7feb300b16a0 msgr2=0x7feb300b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 -- 172.21.15.90:0/3947306555 shutdown_connections 2024-03-31T15:54:15.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 -- 172.21.15.90:0/3947306555 wait complete. 2024-03-31T15:54:15.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 Processor -- start 2024-03-31T15:54:15.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.350+0000 7feb34a00640 1 -- start start 2024-03-31T15:54:15.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.354+0000 7feb34a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 0x7feb30158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb34a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb34a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb34a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feb300cdc30 con 0x7feb300715e0 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb34a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb300cdfa0 con 0x7feb3015a100 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb34a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7feb300ceb40 con 0x7feb300b21c0 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50154/0 (socket says 172.21.15.90:50154) 2024-03-31T15:54:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2de00640 1 -- 172.21.15.90:0/2584832639 learned_addr learned my addr 172.21.15.90:0/2584832639 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2e800640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 0x7feb30158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 -- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 msgr2=0x7feb300c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 -- 172.21.15.90:0/2584832639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 msgr2=0x7feb30158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 0x7feb30158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 -- 172.21.15.90:0/2584832639 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3006e050 con 0x7feb3015a100 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2e800640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 0x7feb30158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2de00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:15.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.358+0000 7feb2f200640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300c9960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb20009dc0 tx=0x7feb2000c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.362+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20019070 con 0x7feb3015a100 2024-03-31T15:54:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.362+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb30141080 con 0x7feb3015a100 2024-03-31T15:54:15.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.362+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb30140750 con 0x7feb3015a100 2024-03-31T15:54:15.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.366+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb200078f0 con 0x7feb3015a100 2024-03-31T15:54:15.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.366+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20007ce0 con 0x7feb3015a100 2024-03-31T15:54:15.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.366+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb300685f0 con 0x7feb3015a100 2024-03-31T15:54:15.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.366+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feb2001c070 con 0x7feb3015a100 2024-03-31T15:54:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.374+0000 7feb26a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 0x7feb10073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.374+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7feb200889e0 con 0x7feb3015a100 2024-03-31T15:54:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.374+0000 7feb2e800640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 0x7feb10073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:15.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.374+0000 7feb2e800640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 0x7feb10073050 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb280af000 tx=0x7feb280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:15.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.378+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7feb200584e0 con 0x7feb3015a100 2024-03-31T15:54:15.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.534+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc60c036a60 con 0x7fc60c0df930 2024-03-31T15:54:15.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.534+0000 7fc60a000640 1 -- 172.21.15.90:0/1887728571 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc6000fdee0 con 0x7fc60c0df930 2024-03-31T15:54:15.531 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 msgr2=0x7fc5f406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 0x7fc5f406a890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc604000f70 tx=0x7fc6040079c0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 msgr2=0x7fc60c0de0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc6000b0000 tx=0x7fc60001e960 comp rx=0 tx=0).stop 2024-03-31T15:54:15.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 shutdown_connections 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc5f40683d0 0x7fc5f406a890 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc604000f70 tx=0x7fc6040079c0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc60c0df930 0x7fc60c0de0b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc60c0529b0 0x7fc60c054290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 --2- 172.21.15.90:0/1887728571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc60c034850 0x7fc60c052fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 >> 172.21.15.90:0/1887728571 conn(0x7fc60c01e6d0 msgr2=0x7fc60c03aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 shutdown_connections 2024-03-31T15:54:15.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.542+0000 7fc613200640 1 -- 172.21.15.90:0/1887728571 wait complete. 2024-03-31T15:54:15.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.554+0000 7f53f9600640 1 -- 172.21.15.90:0/1178694764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f54040c5440 con 0x7f540415a100 2024-03-31T15:54:15.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.554+0000 7f53fbe00640 1 -- 172.21.15.90:0/1178694764 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53f00ad020 con 0x7f540415a100 2024-03-31T15:54:15.551 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-03-31T15:54:15.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 msgr2=0x7f53e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 0x7f53e406a840 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00b450 tx=0x7f53fc00d040 comp rx=0 tx=0).stop 2024-03-31T15:54:15.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 msgr2=0x7f54040c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 0x7f54040c93e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f53f00aa000 tx=0x7f53f001afe0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 shutdown_connections 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f53e4068380 0x7f53e406a840 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00b450 tx=0x7f53fc00d040 comp rx=0 tx=0).stop 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f540415a100 0x7f54040c93e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f54040ca120 0x7f54041fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 --2- 172.21.15.90:0/1178694764 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f54040715e0 0x7f54041fb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 >> 172.21.15.90:0/1178694764 conn(0x7f54040b16a0 msgr2=0x7f54040c4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 shutdown_connections 2024-03-31T15:54:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.562+0000 7f5409a00640 1 -- 172.21.15.90:0/1178694764 wait complete. 2024-03-31T15:54:15.570 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.0 2024-03-31T15:54:15.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.590+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff54c073090 con 0x7ff54c0b21c0 2024-03-31T15:54:15.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.590+0000 7ff542a00640 1 -- 172.21.15.90:0/990439610 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff544057220 con 0x7ff54c0b21c0 2024-03-31T15:54:15.585 INFO:teuthology.orchestra.run.smithi090.stdout:60129542153 2024-03-31T15:54:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 msgr2=0x7ff52c06a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 0x7ff52c06a940 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff53c0b2000 tx=0x7ff53c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 msgr2=0x7ff54c0c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff54400ccb0 tx=0x7ff54400a660 comp rx=0 tx=0).stop 2024-03-31T15:54:15.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.3 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 shutdown_connections 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff52c068480 0x7ff52c06a940 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff54c0b21c0 0x7ff54c0c9c00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff54c071500 0x7ff54c14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 --2- 172.21.15.90:0/990439610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff54c0710a0 0x7ff54c0c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 >> 172.21.15.90:0/990439610 conn(0x7ff54c0b16a0 msgr2=0x7ff54c0cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 shutdown_connections 2024-03-31T15:54:15.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.594+0000 7ff54be00640 1 -- 172.21.15.90:0/990439610 wait complete. 2024-03-31T15:54:15.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.614+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff08006ead0 con 0x7ff080071500 2024-03-31T15:54:15.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.614+0000 7ff076a00640 1 -- 172.21.15.90:0/2439132162 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff070059f40 con 0x7ff080071500 2024-03-31T15:54:15.612 INFO:teuthology.orchestra.run.smithi090.stdout:55834574856 2024-03-31T15:54:15.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 msgr2=0x7ff06006aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 0x7ff06006aa40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0b2000 tx=0x7ff06c0aa070 comp rx=0 tx=0).stop 2024-03-31T15:54:15.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 msgr2=0x7ff08015cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c450 tx=0x7ff07000c910 comp rx=0 tx=0).stop 2024-03-31T15:54:15.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 shutdown_connections 2024-03-31T15:54:15.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff060068580 0x7ff06006aa40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0800c2120 0x7ff0800a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff080071500 0x7ff08015cf80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 --2- 172.21.15.90:0/2439132162 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0800710a0 0x7ff08016bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 >> 172.21.15.90:0/2439132162 conn(0x7ff0800b16a0 msgr2=0x7ff0800c38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 shutdown_connections 2024-03-31T15:54:15.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.618+0000 7ff06be00640 1 -- 172.21.15.90:0/2439132162 wait complete. 2024-03-31T15:54:15.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.622+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff394005590 con 0x7ff3b80cf270 2024-03-31T15:54:15.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.622+0000 7ff3ab400640 1 -- 172.21.15.90:0/3865404785 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff3a4056b00 con 0x7ff3b80cf270 2024-03-31T15:54:15.619 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 msgr2=0x7ff39806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 0x7ff39806a8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000ab90 tx=0x7ff3b0009230 comp rx=0 tx=0).stop 2024-03-31T15:54:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 msgr2=0x7ff3b815e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff3a400e810 tx=0x7ff3a400ecd0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 shutdown_connections 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff398068400 0x7ff39806a8c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3b815c350 0x7ff3b815eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff3b80cf270 0x7ff3b815e560 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 --2- 172.21.15.90:0/3865404785 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff3b80ce780 0x7ff3b8143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 >> 172.21.15.90:0/3865404785 conn(0x7ff3b80d1e20 msgr2=0x7ff3b80d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 shutdown_connections 2024-03-31T15:54:15.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.626+0000 7ff3be200640 1 -- 172.21.15.90:0/3865404785 wait complete. 2024-03-31T15:54:15.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.2 2024-03-31T15:54:15.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.7 2024-03-31T15:54:15.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.1 2024-03-31T15:54:15.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.658+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb9900d1b40 con 0x7fb990142760 2024-03-31T15:54:15.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.658+0000 7fb987400640 1 -- 172.21.15.90:0/3874163583 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb988056c90 con 0x7fb990142760 2024-03-31T15:54:15.653 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:15.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 msgr2=0x7fb96c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 0x7fb96c06a910 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb9900ced50 tx=0x7fb980008040 comp rx=0 tx=0).stop 2024-03-31T15:54:15.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 msgr2=0x7fb99015caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb98800be20 tx=0x7fb98800bf10 comp rx=0 tx=0).stop 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 shutdown_connections 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb96c068450 0x7fb96c06a910 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb990142760 0x7fb99015caf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb990144330 0x7fb99015c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 --2- 172.21.15.90:0/3874163583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9900d0fd0 0x7fb990142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 >> 172.21.15.90:0/3874163583 conn(0x7fb99006c700 msgr2=0x7fb9900ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 shutdown_connections 2024-03-31T15:54:15.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.662+0000 7fb995200640 1 -- 172.21.15.90:0/3874163583 wait complete. 2024-03-31T15:54:15.676 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.4 2024-03-31T15:54:15.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff954068660 con 0x7ff95406cc50 2024-03-31T15:54:15.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff950c00640 1 -- 172.21.15.90:0/54722607 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff944056680 con 0x7ff95406cc50 2024-03-31T15:54:15.680 INFO:teuthology.orchestra.run.smithi090.stdout:64424509449 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 msgr2=0x7ff93406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 0x7ff93406a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009fd0 tx=0x7ff94c009760 comp rx=0 tx=0).stop 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 msgr2=0x7ff9540cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff93800bfb0 tx=0x7ff9440062d0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 shutdown_connections 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff934068380 0x7ff93406a840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff95406cc50 0x7ff9540cd180 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff954150410 0x7ff95406c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 --2- 172.21.15.90:0/54722607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff938004000 0x7ff9540b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 >> 172.21.15.90:0/54722607 conn(0x7ff9540d40f0 msgr2=0x7ff9540b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 shutdown_connections 2024-03-31T15:54:15.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.686+0000 7ff95b800640 1 -- 172.21.15.90:0/54722607 wait complete. 2024-03-31T15:54:15.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.6 2024-03-31T15:54:15.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7feb3006ead0 con 0x7feb3015a100 2024-03-31T15:54:15.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb26a00640 1 -- 172.21.15.90:0/2584832639 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feb200770d0 con 0x7feb3015a100 2024-03-31T15:54:15.729 INFO:teuthology.orchestra.run.smithi090.stdout:64424509449 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 msgr2=0x7feb10073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 0x7feb10073050 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb280af000 tx=0x7feb280a7070 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 msgr2=0x7feb300c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300c9960 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb20009dc0 tx=0x7feb2000c430 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 shutdown_connections 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7feb10070b90 0x7feb10073050 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3015a100 0x7feb300c9960 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feb300b21c0 0x7feb300c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 --2- 172.21.15.90:0/2584832639 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb300715e0 0x7feb30158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.734+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 >> 172.21.15.90:0/2584832639 conn(0x7feb300b16a0 msgr2=0x7feb30071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.738+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 shutdown_connections 2024-03-31T15:54:15.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:15.738+0000 7feb34a00640 1 -- 172.21.15.90:0/2584832639 wait complete. 2024-03-31T15:54:15.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.5 2024-03-31T15:54:16.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-31T15:54:16.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-31T15:54:16.624 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-31T15:54:16.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-31T15:54:16.646 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-31T15:54:16.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-31T15:54:16.710 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-31T15:54:16.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.734+0000 7f1a37600640 1 Processor -- start 2024-03-31T15:54:16.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.734+0000 7f1a37600640 1 -- start start 2024-03-31T15:54:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a37600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1a301624a0 0x7f1a300d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a37600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a300d1400 con 0x7f1a300d2940 2024-03-31T15:54:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a37600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a300b1a20 con 0x7f1a3015e320 2024-03-31T15:54:16.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a37600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3015e700 con 0x7f1a301624a0 2024-03-31T15:54:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a34c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1a3015e320 0x7f1a30161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45470/0 (socket says 172.21.15.90:45470) 2024-03-31T15:54:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.738+0000 7f1a34c00640 1 -- 172.21.15.90:0/1356321721 learned_addr learned my addr 172.21.15.90:0/1356321721 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2fe00640 1 --2- 172.21.15.90:0/1356321721 >> v2:172.21.15.90:3300/0 conn(0x7f1a301624a0 0x7f1a300d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334587994 0 0) 0x7f1a300b1a20 con 0x7f1a3015e320 2024-03-31T15:54:16.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a18003530 con 0x7f1a3015e320 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2619412481 0 0) 0x7f1a18003530 con 0x7f1a3015e320 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 >> v2:172.21.15.90:3300/0 conn(0x7f1a301624a0 msgr2=0x7f1a300d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 --2- 172.21.15.90:0/1356321721 >> v2:172.21.15.90:3300/0 conn(0x7f1a301624a0 0x7f1a300d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 >> v1:172.21.15.90:6789/0 conn(0x7f1a300d2940 legacy=0x7f1a300b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a300b1d00 con 0x7f1a3015e320 2024-03-31T15:54:16.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f1a24001680 con 0x7f1a3015e320 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 >> v1:172.21.15.102:6789/0 conn(0x7f1a3015e320 legacy=0x7f1a30161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 0x7f1a18003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18004810 0x7f1a18004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a180051d0 0x7f1a1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1800c0a0 con 0x7f1a18004810 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1800c220 con 0x7f1a18003a10 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1800c420 con 0x7f1a180051d0 2024-03-31T15:54:16.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.742+0000 7f1a34c00640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 0x7f1a18003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a35600640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a180051d0 0x7f1a1800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a34c00640 1 -- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a180051d0 msgr2=0x7f1a1800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a34c00640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a180051d0 0x7f1a1800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a34c00640 1 -- 172.21.15.90:0/1356321721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18004810 msgr2=0x7f1a18004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a34c00640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18004810 0x7f1a18004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a34c00640 1 -- 172.21.15.90:0/1356321721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a18003530 con 0x7f1a18003a10 2024-03-31T15:54:16.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.746+0000 7f1a35600640 1 -- 172.21.15.90:0/1356321721 reap_dead start 2024-03-31T15:54:16.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-31T15:54:16.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a34c00640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 0x7f1a18003e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c190 tx=0x7f1a2401f330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f0ba83090a4b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007c90 con 0x7f1a18003a10 2024-03-31T15:54:16.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a24005b00 con 0x7f1a18003a10 2024-03-31T15:54:16.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a2f400640 1 -- 172.21.15.90:0/1356321721 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007070 con 0x7f1a18003a10 2024-03-31T15:54:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 -- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 msgr2=0x7f1a18003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 0x7f1a18003e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c190 tx=0x7f1a2401f330 comp rx=0 tx=0).stop 2024-03-31T15:54:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 -- 172.21.15.90:0/1356321721 shutdown_connections 2024-03-31T15:54:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 --2- 172.21.15.90:0/1356321721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a18003a10 0x7f1a18003e10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 -- 172.21.15.90:0/1356321721 >> 172.21.15.90:0/1356321721 conn(0x7f1a3008bfd0 msgr2=0x7f1a300d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:16.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 -- 172.21.15.90:0/1356321721 shutdown_connections 2024-03-31T15:54:16.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.762+0000 7f1a37600640 1 -- 172.21.15.90:0/1356321721 wait complete. 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 Processor -- start 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 -- start start 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a300c85d0 0x7f1a3015a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a300b3d10 0x7f1a300b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300d0d20 con 0x7f1a18003a10 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1a300d2db0 con 0x7f1a300b3d10 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a37600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1a300d0ea0 con 0x7f1a300c85d0 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a34c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a34c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51516/0 (socket says 172.21.15.90:51516) 2024-03-31T15:54:16.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a34c00640 1 -- 172.21.15.90:0/2894336337 learned_addr learned my addr 172.21.15.90:0/2894336337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.766+0000 7f1a35600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a300b3d10 0x7f1a300b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 -- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a300c85d0 msgr2=0x7f1a3015a2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-31T15:54:16.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a300c85d0 0x7f1a3015a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 -- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a300b3d10 msgr2=0x7f1a300b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a300b3d10 0x7f1a300b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 -- 172.21.15.90:0/2894336337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a300c6520 con 0x7f1a18003a10 2024-03-31T15:54:16.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a34c00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c190 tx=0x7f1a18005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a240049f0 con 0x7f1a18003a10 2024-03-31T15:54:16.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a24004b90 con 0x7f1a18003a10 2024-03-31T15:54:16.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.770+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24007070 con 0x7f1a18003a10 2024-03-31T15:54:16.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.774+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a300c6800 con 0x7f1a18003a10 2024-03-31T15:54:16.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.774+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a300c6dc0 con 0x7f1a18003a10 2024-03-31T15:54:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1a240c1060 con 0x7f1a18003a10 2024-03-31T15:54:16.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a30068660 con 0x7f1a18003a10 2024-03-31T15:54:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a2cc00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 0x7f1a1006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1a24124370 con 0x7f1a18003a10 2024-03-31T15:54:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a2fe00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 0x7f1a1006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.782+0000 7f1a2fe00640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 0x7f1a1006a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006e90 tx=0x7f1a1c006de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.802+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1a240b6040 con 0x7f1a18003a10 2024-03-31T15:54:16.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 Processor -- start 2024-03-31T15:54:16.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 -- start start 2024-03-31T15:54:16.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f121c06d9f0 0x7f121c0ca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121c0cacd0 con 0x7f121c0ca8c0 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121c15b2e0 con 0x7f121c1511b0 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.818+0000 7f1223600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f121c06cfa0 con 0x7f121c06d9f0 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f121c0ca8c0 0x7f121c0cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33964/0 (socket says 172.21.15.90:33964) 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1220c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f121c1511b0 0x7f121c15a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45474/0 (socket says 172.21.15.90:45474) 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 -- 172.21.15.90:0/266921818 learned_addr learned my addr 172.21.15.90:0/266921818 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:16.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121be00640 1 --2- 172.21.15.90:0/266921818 >> v2:172.21.15.102:3300/0 conn(0x7f121c06d9f0 0x7f121c0ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310081995 0 0) 0x7f121c15b2e0 con 0x7f121c1511b0 2024-03-31T15:54:16.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1200003530 con 0x7f121c1511b0 2024-03-31T15:54:16.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2930773788 0 0) 0x7f1200003530 con 0x7f121c1511b0 2024-03-31T15:54:16.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 >> v2:172.21.15.102:3300/0 conn(0x7f121c06d9f0 msgr2=0x7f121c0ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 --2- 172.21.15.90:0/266921818 >> v2:172.21.15.102:3300/0 conn(0x7f121c06d9f0 0x7f121c0ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 >> v1:172.21.15.90:6789/0 conn(0x7f121c0ca8c0 legacy=0x7f121c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c15b5c0 con 0x7f121c1511b0 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f120c001680 con 0x7f121c1511b0 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 >> v1:172.21.15.102:6789/0 conn(0x7f121c1511b0 legacy=0x7f121c15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1200003a10 0x7f1200003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1200004810 0x7f1200004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 0x7f120000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f121c06cfa0 con 0x7f1200004810 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f120000c0d0 con 0x7f12000051d0 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f120000c2d0 con 0x7f1200003a10 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f121be00640 1 --2- 172.21.15.90:0/266921818 >> v2:172.21.15.102:3300/0 conn(0x7f121c06d9f0 0x7f121c0ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 0x7f120000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1220c00640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1200003a10 0x7f1200003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 -- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1200003a10 msgr2=0x7f1200003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1200003a10 0x7f1200003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 -- 172.21.15.90:0/266921818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1200004810 msgr2=0x7f1200004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1200004810 0x7f1200004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.822+0000 7f1221600640 1 -- 172.21.15.90:0/266921818 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1200003530 con 0x7f12000051d0 2024-03-31T15:54:16.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.826+0000 7f1221600640 1 -- 172.21.15.90:0/266921818 reap_dead start 2024-03-31T15:54:16.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1221600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 0x7f120000b9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1208001820 tx=0x7f120801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72ff966307d7dfd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208001470 con 0x7f12000051d0 2024-03-31T15:54:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1208004ad0 con 0x7f12000051d0 2024-03-31T15:54:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f121b400640 1 -- 172.21.15.90:0/266921818 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208007070 con 0x7f12000051d0 2024-03-31T15:54:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 -- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 msgr2=0x7f120000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 0x7f120000b9d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1208001820 tx=0x7f120801c2e0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 -- 172.21.15.90:0/266921818 shutdown_connections 2024-03-31T15:54:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 --2- 172.21.15.90:0/266921818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12000051d0 0x7f120000b9d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 -- 172.21.15.90:0/266921818 >> 172.21.15.90:0/266921818 conn(0x7f121c0b1880 msgr2=0x7f121c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:16.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 -- 172.21.15.90:0/266921818 shutdown_connections 2024-03-31T15:54:16.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.834+0000 7f1223600640 1 -- 172.21.15.90:0/266921818 wait complete. 2024-03-31T15:54:16.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 Processor -- start 2024-03-31T15:54:16.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 -- start start 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 0x7f121c2007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 0x7f121c0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f121c0c4200 0x7f121c0b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f121c06ddd0 con 0x7f12000051d0 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f121c06d4d0 con 0x7f121c0c4200 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1223600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f121c0cf6d0 con 0x7f121c0c38d0 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1220c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 0x7f121c2007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1220c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 0x7f121c2007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51526/0 (socket says 172.21.15.90:51526) 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1220c00640 1 -- 172.21.15.90:0/4201968772 learned_addr learned my addr 172.21.15.90:0/4201968772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f121be00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 0x7f121c0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.838+0000 7f1221600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f121c0c4200 0x7f121c0b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 -- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f121c0c4200 msgr2=0x7f121c0b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f121c0c4200 0x7f121c0b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 -- 172.21.15.90:0/4201968772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 msgr2=0x7f121c2007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 0x7f121c2007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 -- 172.21.15.90:0/4201968772 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c2051e0 con 0x7f121c0c38d0 2024-03-31T15:54:16.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f121be00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 0x7f121c0b0840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f121c069d00 tx=0x7f121400c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121401b070 con 0x7f121c0c38d0 2024-03-31T15:54:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121400ce50 con 0x7f121c0c38d0 2024-03-31T15:54:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.842+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12140166b0 con 0x7f121c0c38d0 2024-03-31T15:54:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.846+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121c1517e0 con 0x7f121c0c38d0 2024-03-31T15:54:16.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.846+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f121c151da0 con 0x7f121c0c38d0 2024-03-31T15:54:16.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.846+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f121401e070 con 0x7f121c0c38d0 2024-03-31T15:54:16.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.850+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121c068660 con 0x7f121c0c38d0 2024-03-31T15:54:16.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.850+0000 7f1218c00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 0x7f11fc06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.850+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1214086ba0 con 0x7f121c0c38d0 2024-03-31T15:54:16.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.850+0000 7f1220c00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 0x7f11fc06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.850+0000 7f1220c00640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 0x7f11fc06a7e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f120c001c10 tx=0x7f120c000fb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.858+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1214056860 con 0x7f121c0c38d0 2024-03-31T15:54:16.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 Processor -- start 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 -- start start 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f996806d9f0 0x7f99680ca380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 0x7f99680cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99680cacd0 con 0x7f99681511b0 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f996806cfa0 con 0x7f99680ca8c0 2024-03-31T15:54:16.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f9970600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f996806de50 con 0x7f996806d9f0 2024-03-31T15:54:16.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f996806d9f0 0x7f99680ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f996806d9f0 0x7f99680ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47238/0 (socket says 172.21.15.90:47238) 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 -- 172.21.15.90:0/3334529706 learned_addr learned my addr 172.21.15.90:0/3334529706 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 -- 172.21.15.90:0/3334529706 >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 msgr2=0x7f99680cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996e600640 1 --2- 172.21.15.90:0/3334529706 >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 0x7f99680cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 --2- 172.21.15.90:0/3334529706 >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 0x7f99680cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.914+0000 7f996d200640 1 -- 172.21.15.90:0/3334529706 >> v1:172.21.15.90:6789/0 conn(0x7f99681511b0 legacy=0x7f996815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.918+0000 7f996d200640 1 -- 172.21.15.90:0/3334529706 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996815b2e0 con 0x7f996806d9f0 2024-03-31T15:54:16.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.918+0000 7f996e600640 1 --2- 172.21.15.90:0/3334529706 >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 0x7f99680cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:16.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f996d200640 1 --2- 172.21.15.90:0/3334529706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996806d9f0 0x7f99680ca380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f99680b28e0 tx=0x7f995401cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c856ba6bca7a438f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f9967e00640 1 -- 172.21.15.90:0/3334529706 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540b2070 con 0x7f996806d9f0 2024-03-31T15:54:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f9967e00640 1 -- 172.21.15.90:0/3334529706 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99540a5b30 con 0x7f996806d9f0 2024-03-31T15:54:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f9967e00640 1 -- 172.21.15.90:0/3334529706 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540ad440 con 0x7f996806d9f0 2024-03-31T15:54:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f627d200640 1 Processor -- start 2024-03-31T15:54:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f9970600640 1 -- 172.21.15.90:0/3334529706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996806d9f0 msgr2=0x7f99680ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f9970600640 1 --2- 172.21.15.90:0/3334529706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996806d9f0 0x7f99680ca380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f99680b28e0 tx=0x7f995401cb70 comp rx=0 tx=0).stop 2024-03-31T15:54:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 -- 172.21.15.90:0/3334529706 shutdown_connections 2024-03-31T15:54:16.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 --2- 172.21.15.90:0/3334529706 >> v2:172.21.15.90:3300/0 conn(0x7f99680ca8c0 0x7f99680cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 --2- 172.21.15.90:0/3334529706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996806d9f0 0x7f99680ca380 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 -- 172.21.15.90:0/3334529706 >> 172.21.15.90:0/3334529706 conn(0x7f99680b1880 msgr2=0x7f99680b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.922+0000 7f627d200640 1 -- start start 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f627d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6278072c30 0x7f62780cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f627d200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f627806ed80 0x7f62780cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f627d200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6278166310 con 0x7f6278071220 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f627d200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f62781665f0 con 0x7f6278072c30 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f627d200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f62781668d0 con 0x7f627806ed80 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6275e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6278072c30 0x7f62780cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6276800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6278071220 0x7f6278161c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45484/0 (socket says 172.21.15.90:45484) 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6276800640 1 -- 172.21.15.90:0/3611688845 learned_addr learned my addr 172.21.15.90:0/3611688845 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:16.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6275e00640 1 -- 172.21.15.90:0/3611688845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f627806ed80 msgr2=0x7f62780cc740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6275e00640 1 --2- 172.21.15.90:0/3611688845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f627806ed80 0x7f62780cc740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6275e00640 1 -- 172.21.15.90:0/3611688845 >> v1:172.21.15.102:6789/0 conn(0x7f6278071220 legacy=0x7f6278161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f6275e00640 1 -- 172.21.15.90:0/3611688845 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6278166bb0 con 0x7f6278072c30 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f29bd400640 1 Processor -- start 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f29bd400640 1 -- start start 2024-03-31T15:54:16.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 -- 172.21.15.90:0/3334529706 shutdown_connections 2024-03-31T15:54:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29bd400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29b80b8310 0x7f29b816d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29bd400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b8162db0 con 0x7f29b80b53f0 2024-03-31T15:54:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29bd400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b80b6e50 con 0x7f29b80d4ba0 2024-03-31T15:54:16.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29bd400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f29b80d1600 con 0x7f29b80b8310 2024-03-31T15:54:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.926+0000 7f9970600640 1 -- 172.21.15.90:0/3334529706 wait complete. 2024-03-31T15:54:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f9970600640 1 Processor -- start 2024-03-31T15:54:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29b7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f29b80b53f0 0x7f29b80d07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:33984/0 (socket says 172.21.15.90:33984) 2024-03-31T15:54:16.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29b7e00640 1 -- 172.21.15.90:0/117951184 learned_addr learned my addr 172.21.15.90:0/117951184 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:16.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29bca00640 1 --2- 172.21.15.90:0/117951184 >> v2:172.21.15.90:3300/0 conn(0x7f29b80b8310 0x7f29b816d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f6275e00640 1 --2- 172.21.15.90:0/3611688845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6278072c30 0x7f62780cbe00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62680097c0 tx=0x7f626801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8313a99ff37df899 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29b6a00640 1 -- 172.21.15.90:0/117951184 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216407164 0 0) 0x7f29b8162db0 con 0x7f29b80b53f0 2024-03-31T15:54:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.930+0000 7f29b6a00640 1 -- 172.21.15.90:0/117951184 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f299c003530 con 0x7f29b80b53f0 2024-03-31T15:54:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f29bca00640 1 -- 172.21.15.90:0/117951184 >> v1:172.21.15.102:6789/0 conn(0x7f29b80d4ba0 legacy=0x7f29b80d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f9970600640 1 -- start start 2024-03-31T15:54:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f29bca00640 1 -- 172.21.15.90:0/117951184 >> v1:172.21.15.90:6789/0 conn(0x7f29b80b53f0 legacy=0x7f29b80d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f29bca00640 1 -- 172.21.15.90:0/117951184 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b80cf6b0 con 0x7f29b80b8310 2024-03-31T15:54:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f6275400640 1 -- 172.21.15.90:0/3611688845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680b0070 con 0x7f6278072c30 2024-03-31T15:54:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f6275400640 1 -- 172.21.15.90:0/3611688845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62680ab600 con 0x7f6278072c30 2024-03-31T15:54:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f6275400640 1 -- 172.21.15.90:0/3611688845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680ab9d0 con 0x7f6278072c30 2024-03-31T15:54:16.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f9970600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996806d9f0 0x7f9968150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 -- 172.21.15.90:0/3611688845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6278072c30 msgr2=0x7f62780cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 --2- 172.21.15.90:0/3611688845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6278072c30 0x7f62780cbe00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62680097c0 tx=0x7f626801c350 comp rx=0 tx=0).stop 2024-03-31T15:54:16.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 -- 172.21.15.90:0/3611688845 shutdown_connections 2024-03-31T15:54:16.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 --2- 172.21.15.90:0/3611688845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f627806ed80 0x7f62780cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 --2- 172.21.15.90:0/3611688845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6278072c30 0x7f62780cbe00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.934+0000 7f627d200640 1 -- 172.21.15.90:0/3611688845 >> 172.21.15.90:0/3611688845 conn(0x7f62780b1a50 msgr2=0x7f62780b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:16.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.938+0000 7f29bca00640 1 --2- 172.21.15.90:0/117951184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80b8310 0x7f29b816d4c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097c0 tx=0x7f29a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9b84a94fd8dc88 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.938+0000 7f29b6a00640 1 -- 172.21.15.90:0/117951184 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a00b1070 con 0x7f29b80b8310 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.938+0000 7f29b6a00640 1 -- 172.21.15.90:0/117951184 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a00ac5f0 con 0x7f29b80b8310 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.938+0000 7f29b6a00640 1 -- 172.21.15.90:0/117951184 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a00b1070 con 0x7f29b80b8310 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f9970600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f9970600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99681511b0 0x7f9968156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f9970600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f996815af30 con 0x7f996806d9f0 2024-03-31T15:54:16.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f9970600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99680cf6d0 con 0x7f99681511b0 2024-03-31T15:54:16.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f9970600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9968157df0 con 0x7f99680ca8c0 2024-03-31T15:54:16.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f627d200640 1 -- 172.21.15.90:0/3611688845 shutdown_connections 2024-03-31T15:54:16.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f627d200640 1 -- 172.21.15.90:0/3611688845 wait complete. 2024-03-31T15:54:16.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 -- 172.21.15.90:0/117951184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80b8310 msgr2=0x7f29b816d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 --2- 172.21.15.90:0/117951184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80b8310 0x7f29b816d4c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f29a00097c0 tx=0x7f29a001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50220/0 (socket says 172.21.15.90:50220) 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 -- 172.21.15.90:0/1221712074 learned_addr learned my addr 172.21.15.90:0/1221712074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996dc00640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996806d9f0 0x7f9968150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996e600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99681511b0 0x7f9968156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 -- 172.21.15.90:0/117951184 shutdown_connections 2024-03-31T15:54:16.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 --2- 172.21.15.90:0/117951184 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80b8310 0x7f29b816d4c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 -- 172.21.15.90:0/117951184 >> 172.21.15.90:0/117951184 conn(0x7f29b80d3f60 msgr2=0x7f29b80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 -- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99681511b0 msgr2=0x7f9968156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99681511b0 0x7f9968156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 -- 172.21.15.90:0/1221712074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996806d9f0 msgr2=0x7f9968150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996806d9f0 0x7f9968150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 -- 172.21.15.90:0/1221712074 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99681f3e80 con 0x7f99680ca8c0 2024-03-31T15:54:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 -- 172.21.15.90:0/117951184 shutdown_connections 2024-03-31T15:54:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f29bd400640 1 -- 172.21.15.90:0/117951184 wait complete. 2024-03-31T15:54:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f627d200640 1 Processor -- start 2024-03-31T15:54:16.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f996d200640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f99540b0040 tx=0x7f9954019910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.942+0000 7f627d200640 1 -- start start 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540bd070 con 0x7f99680ca8c0 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99540092d0 con 0x7f99680ca8c0 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540b2070 con 0x7f99680ca8c0 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99681f4160 con 0x7f99680ca8c0 2024-03-31T15:54:16.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9968156740 con 0x7f99680ca8c0 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 0x7f62781513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6278071220 0x7f6278150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 0x7f62780cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62780cc4c0 con 0x7f627806ed80 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62780cc340 con 0x7f6278071220 2024-03-31T15:54:16.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f627d200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6278073010 con 0x7f6278072c30 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f626fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6278071220 0x7f6278150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6275e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 0x7f62781513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6275e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 0x7f62781513a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51564/0 (socket says 172.21.15.90:51564) 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6275e00640 1 -- 172.21.15.90:0/983615849 learned_addr learned my addr 172.21.15.90:0/983615849 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 0x7f62780cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 -- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6278071220 msgr2=0x7f6278150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6278071220 0x7f6278150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 -- 172.21.15.90:0/983615849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 msgr2=0x7f62781513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 0x7f62781513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 -- 172.21.15.90:0/983615849 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62780ca090 con 0x7f6278072c30 2024-03-31T15:54:16.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f29bd400640 1 Processor -- start 2024-03-31T15:54:16.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9968068660 con 0x7f99680ca8c0 2024-03-31T15:54:16.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f6276800640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 0x7f62780cd3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f627000e450 tx=0x7f627000e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.946+0000 7f29bd400640 1 -- start start 2024-03-31T15:54:16.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f99540c0070 con 0x7f99680ca8c0 2024-03-31T15:54:16.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f9965600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 0x7f994806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f99540b9e30 con 0x7f99680ca8c0 2024-03-31T15:54:16.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f996dc00640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 0x7f994806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29b80b8310 0x7f29b80d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80d4ba0 0x7f29b8160be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29b7e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29b7e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50242/0 (socket says 172.21.15.90:50242) 2024-03-31T15:54:16.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29b80d2de0 con 0x7f29b80d4ba0 2024-03-31T15:54:16.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f996dc00640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 0x7f994806a890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f99680c9f60 tx=0x7f995c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29b7e00640 1 -- 172.21.15.90:0/1886338909 learned_addr learned my addr 172.21.15.90:0/1886338909 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f29b80cff50 con 0x7f29b80b8310 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f29b8170630 con 0x7f29b80b53f0 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29b7400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29b80b8310 0x7f29b80d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270009480 con 0x7f6278072c30 2024-03-31T15:54:16.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6270018070 con 0x7f6278072c30 2024-03-31T15:54:16.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270009820 con 0x7f6278072c30 2024-03-31T15:54:16.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.950+0000 7f29bca00640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80d4ba0 0x7f29b8160be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62780c5200 con 0x7f6278072c30 2024-03-31T15:54:16.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62780c54a0 con 0x7f6278072c30 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 -- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29b80b8310 msgr2=0x7f29b80d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29b80b8310 0x7f29b80d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 -- 172.21.15.90:0/1886338909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80d4ba0 msgr2=0x7f29b8160be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80d4ba0 0x7f29b8160be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b81607e0 con 0x7f29b80b53f0 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f29b7e00640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c420 tx=0x7f29a800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6278068660 con 0x7f6278072c30 2024-03-31T15:54:16.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.954+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f99680cacd0 con 0x7f99680ca8c0 2024-03-31T15:54:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.958+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6270012070 con 0x7f6278072c30 2024-03-31T15:54:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.958+0000 7f626e000640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 0x7f625406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.958+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f627008a2d0 con 0x7f6278072c30 2024-03-31T15:54:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.958+0000 7f6275e00640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 0x7f625406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6270010040 con 0x7f6278072c30 2024-03-31T15:54:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8017070 con 0x7f29b80b53f0 2024-03-31T15:54:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a8005cf0 con 0x7f29b80b53f0 2024-03-31T15:54:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80072d0 con 0x7f29b80b53f0 2024-03-31T15:54:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b815f200 con 0x7f29b80b53f0 2024-03-31T15:54:16.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b815faa0 con 0x7f29b80b53f0 2024-03-31T15:54:16.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f6275e00640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 0x7f625406a910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62680097c0 tx=0x7f62680a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.962+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29a801a070 con 0x7f29b80b53f0 2024-03-31T15:54:16.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f29afe00640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 0x7f299406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f29b7400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 0x7f299406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f29a8053cf0 con 0x7f29b80b53f0 2024-03-31T15:54:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f29b7400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 0x7f299406a890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f29b00012f0 tx=0x7f29b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.966+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b8068d00 con 0x7f29b80b53f0 2024-03-31T15:54:16.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.974+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f29b81607e0 con 0x7f29b80b53f0 2024-03-31T15:54:16.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.982+0000 7f5816000640 1 Processor -- start 2024-03-31T15:54:16.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.982+0000 7f5816000640 1 -- start start 2024-03-31T15:54:16.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f5816000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 0x7f581006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f5816000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581015b2e0 con 0x7f581015a820 2024-03-31T15:54:16.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f5816000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f581015b5c0 con 0x7f58100caa90 2024-03-31T15:54:16.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f5816000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f581006cfa0 con 0x7f58101511b0 2024-03-31T15:54:16.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f580f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 0x7f581006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f580fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f58100caa90 0x7f58100cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:45512/0 (socket says 172.21.15.90:45512) 2024-03-31T15:54:16.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f580fe00640 1 -- 172.21.15.90:0/1729641599 learned_addr learned my addr 172.21.15.90:0/1729641599 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:16.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.986+0000 7f580e000640 1 -- 172.21.15.90:0/1729641599 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2109530214 0 0) 0x7f581015b5c0 con 0x7f58100caa90 2024-03-31T15:54:16.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.990+0000 7f580e000640 1 -- 172.21.15.90:0/1729641599 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57f4003530 con 0x7f58100caa90 2024-03-31T15:54:16.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.990+0000 7f580f400640 1 -- 172.21.15.90:0/1729641599 >> v1:172.21.15.102:6789/0 conn(0x7f58100caa90 legacy=0x7f58100cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.990+0000 7f580f400640 1 -- 172.21.15.90:0/1729641599 >> v1:172.21.15.90:6789/0 conn(0x7f581015a820 legacy=0x7f58100ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.990+0000 7f580f400640 1 -- 172.21.15.90:0/1729641599 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581006d1f0 con 0x7f58101511b0 2024-03-31T15:54:16.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 Processor -- start 2024-03-31T15:54:16.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 -- start start 2024-03-31T15:54:16.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f580f400640 1 --2- 172.21.15.90:0/1729641599 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 0x7f581006d9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5800007200 tx=0x7f580001d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575f332ff8b7bc18 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:16.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f580e000640 1 -- 172.21.15.90:0/1729641599 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58000ad070 con 0x7f58101511b0 2024-03-31T15:54:16.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f580e000640 1 -- 172.21.15.90:0/1729641599 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5800008be0 con 0x7f58101511b0 2024-03-31T15:54:16.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f580e000640 1 -- 172.21.15.90:0/1729641599 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580000a0d0 con 0x7f58101511b0 2024-03-31T15:54:16.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f24c40cc990 0x7f24c4162650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24c40cd6a0 0x7f24c40cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:16.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c40adbf0 con 0x7f24c415d760 2024-03-31T15:54:16.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f24c40ce050 con 0x7f24c40cc990 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c3e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24c4162b90 con 0x7f24c40cd6a0 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24c40cd6a0 0x7f24c40cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24c40cd6a0 0x7f24c40cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47262/0 (socket says 172.21.15.90:47262) 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 -- 172.21.15.90:0/3583504185 learned_addr learned my addr 172.21.15.90:0/3583504185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2a00640 1 --2- 172.21.15.90:0/3583504185 >> v2:172.21.15.90:3300/0 conn(0x7f24c40cc990 0x7f24c4162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:16.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 -- 172.21.15.90:0/3583504185 >> v2:172.21.15.90:3300/0 conn(0x7f24c40cc990 msgr2=0x7f24c4162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:16.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.002+0000 7f5816000640 1 -- 172.21.15.90:0/1729641599 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 msgr2=0x7f581006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.002+0000 7f5816000640 1 --2- 172.21.15.90:0/1729641599 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 0x7f581006d9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5800007200 tx=0x7f580001d4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 --2- 172.21.15.90:0/3583504185 >> v2:172.21.15.90:3300/0 conn(0x7f24c40cc990 0x7f24c4162650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:16.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 -- 172.21.15.90:0/3583504185 >> v1:172.21.15.102:6789/0 conn(0x7f24c415d760 legacy=0x7f24c415dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:16.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:16.998+0000 7f24c2000640 1 -- 172.21.15.90:0/3583504185 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c40aded0 con 0x7f24c40cd6a0 2024-03-31T15:54:17.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 -- 172.21.15.90:0/1729641599 shutdown_connections 2024-03-31T15:54:17.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 --2- 172.21.15.90:0/1729641599 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58101511b0 0x7f581006d9f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 -- 172.21.15.90:0/1729641599 >> 172.21.15.90:0/1729641599 conn(0x7f58100b1880 msgr2=0x7f58100b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f24c2000640 1 --2- 172.21.15.90:0/3583504185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40cdb10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24c40b2760 tx=0x7f24ac01ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3b8c7d081ad3fe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 -- 172.21.15.90:0/1729641599 shutdown_connections 2024-03-31T15:54:17.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 -- 172.21.15.90:0/1729641599 wait complete. 2024-03-31T15:54:17.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f5816000640 1 Processor -- start 2024-03-31T15:54:17.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f24c1600640 1 -- 172.21.15.90:0/3583504185 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0b5070 con 0x7f24c40cd6a0 2024-03-31T15:54:17.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f24c1600640 1 -- 172.21.15.90:0/3583504185 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ac0a8a10 con 0x7f24c40cd6a0 2024-03-31T15:54:17.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.006+0000 7f24c1600640 1 -- 172.21.15.90:0/3583504185 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0a8de0 con 0x7f24c40cd6a0 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 -- 172.21.15.90:0/3583504185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 msgr2=0x7f24c40cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 --2- 172.21.15.90:0/3583504185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40cdb10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24c40b2760 tx=0x7f24ac01ed10 comp rx=0 tx=0).stop 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 -- 172.21.15.90:0/3583504185 shutdown_connections 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 --2- 172.21.15.90:0/3583504185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40cdb10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 --2- 172.21.15.90:0/3583504185 >> v2:172.21.15.90:3300/0 conn(0x7f24c40cc990 0x7f24c4162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 -- 172.21.15.90:0/3583504185 >> 172.21.15.90:0/3583504185 conn(0x7f24c40b16a0 msgr2=0x7f24c40b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 -- 172.21.15.90:0/3583504185 shutdown_connections 2024-03-31T15:54:17.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 -- 172.21.15.90:0/3583504185 wait complete. 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f24c3e00640 1 Processor -- start 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.010+0000 7f5816000640 1 -- start start 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58100caa90 0x7f581014e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 0x7f58100cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58100ce010 con 0x7f58100caa90 2024-03-31T15:54:17.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f581015b090 con 0x7f58101511b0 2024-03-31T15:54:17.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5816000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f58100ce190 con 0x7f581015a820 2024-03-31T15:54:17.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f24c3e00640 1 -- start start 2024-03-31T15:54:17.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 0x7f58100cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47274/0 (socket says 172.21.15.90:47274) 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 -- 172.21.15.90:0/1967741104 learned_addr learned my addr 172.21.15.90:0/1967741104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 0x7f58100cfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50264/0 (socket says 172.21.15.90:50264) 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 -- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 msgr2=0x7f58100cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 0x7f58100cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 -- 172.21.15.90:0/1967741104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58100caa90 msgr2=0x7f581014e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58100caa90 0x7f581014e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 -- 172.21.15.90:0/1967741104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5810159050 con 0x7f58101511b0 2024-03-31T15:54:17.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f580ea00640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c3c0 tx=0x7f57fc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc017070 con 0x7f58101511b0 2024-03-31T15:54:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57fc0052d0 con 0x7f58101511b0 2024-03-31T15:54:17.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc003840 con 0x7f58101511b0 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.014+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 0x7f24c40b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 0x7f24c41f8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f24c40ae490 con 0x7f24c40cc990 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24c40ae800 con 0x7f24c40cd6a0 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f24c40ad8c0 con 0x7f24c415d760 2024-03-31T15:54:17.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5810159330 con 0x7f58101511b0 2024-03-31T15:54:17.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58101598f0 con 0x7f58101511b0 2024-03-31T15:54:17.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5810068660 con 0x7f58101511b0 2024-03-31T15:54:17.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 0x7f24c40b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 0x7f24c40b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51592/0 (socket says 172.21.15.90:51592) 2024-03-31T15:54:17.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c2a00640 1 -- 172.21.15.90:0/3780737355 learned_addr learned my addr 172.21.15.90:0/3780737355 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 0x7f24c41f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 -- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 msgr2=0x7f24c40b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 -- 172.21.15.90:0/3780737355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 msgr2=0x7f24c40b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 0x7f24c40b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 -- 172.21.15.90:0/3780737355 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c415e5c0 con 0x7f24c415d760 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c2000640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3400640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 0x7f24c41f8140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c3f0 tx=0x7f24b400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b40052d0 con 0x7f24c415d760 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b4005470 con 0x7f24c415d760 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4007260 con 0x7f24c415d760 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 -- 172.21.15.90:0/3780737355 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c415e880 con 0x7f24c415d760 2024-03-31T15:54:17.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.018+0000 7f24c3e00640 1 -- 172.21.15.90:0/3780737355 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c415ee40 con 0x7f24c415d760 2024-03-31T15:54:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.022+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f57fc01a070 con 0x7f58101511b0 2024-03-31T15:54:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.022+0000 7f5807400640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 0x7f57f006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.022+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249c0052a0 con 0x7f24c415d760 2024-03-31T15:54:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.022+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f57fc086000 con 0x7f58101511b0 2024-03-31T15:54:17.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.022+0000 7f580f400640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 0x7f57f006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.026+0000 7f580f400640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 0x7f57f006a840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f58000af000 tx=0x7f5800004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.026+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f24b4007400 con 0x7f24c415d760 2024-03-31T15:54:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.026+0000 7f24baa00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 0x7f24a006a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.026+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f24b4085f40 con 0x7f24c415d760 2024-03-31T15:54:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.026+0000 7f24c2a00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 0x7f24a006a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.030+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f57fc005480 con 0x7f58101511b0 2024-03-31T15:54:17.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.030+0000 7f24c2a00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 0x7f24a006a910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009a40 tx=0x7f24bc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.034+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f24b4055b50 con 0x7f24c415d760 2024-03-31T15:54:17.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 Processor -- start 2024-03-31T15:54:17.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 -- start start 2024-03-31T15:54:17.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac166310 con 0x7f8fac0cce50 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fac1665f0 con 0x7f8fac06ed80 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.050+0000 7f8fb1200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8fac1668d0 con 0x7f8fac072c30 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50294/0 (socket says 172.21.15.90:50294) 2024-03-31T15:54:17.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 -- 172.21.15.90:0/438316003 learned_addr learned my addr 172.21.15.90:0/438316003 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 -- 172.21.15.90:0/438316003 >> v1:172.21.15.102:6789/0 conn(0x7f8fac06ed80 legacy=0x7f8fac0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:17.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 -- 172.21.15.90:0/438316003 >> v1:172.21.15.90:6789/0 conn(0x7f8fac0cce50 legacy=0x7f8fac161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:17.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.054+0000 7f8fa9e00640 1 -- 172.21.15.90:0/438316003 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac166bb0 con 0x7f8fac072c30 2024-03-31T15:54:17.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.062+0000 7f8fa9e00640 1 --2- 172.21.15.90:0/438316003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f940098a0 tx=0x7f8f9401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdae4cb805a724f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.062+0000 7f8fa8a00640 1 -- 172.21.15.90:0/438316003 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940aa610 con 0x7f8fac072c30 2024-03-31T15:54:17.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.062+0000 7f8fa8a00640 1 -- 172.21.15.90:0/438316003 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f940af070 con 0x7f8fac072c30 2024-03-31T15:54:17.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.062+0000 7f8fa8a00640 1 -- 172.21.15.90:0/438316003 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940aa9d0 con 0x7f8fac072c30 2024-03-31T15:54:17.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 -- 172.21.15.90:0/438316003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 msgr2=0x7f8fac0cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 --2- 172.21.15.90:0/438316003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8f940098a0 tx=0x7f8f9401c4d0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 -- 172.21.15.90:0/438316003 shutdown_connections 2024-03-31T15:54:17.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 --2- 172.21.15.90:0/438316003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac0cbe00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 -- 172.21.15.90:0/438316003 >> 172.21.15.90:0/438316003 conn(0x7f8fac0b1a50 msgr2=0x7f8fac0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 -- 172.21.15.90:0/438316003 shutdown_connections 2024-03-31T15:54:17.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 -- 172.21.15.90:0/438316003 wait complete. 2024-03-31T15:54:17.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.066+0000 7f8fb1200640 1 Processor -- start 2024-03-31T15:54:17.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fb1200640 1 -- start start 2024-03-31T15:54:17.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fb1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fb1200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac163c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fb1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fac0cce50 0x7f8fac164160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fa9400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fa9400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac163c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50296/0 (socket says 172.21.15.90:50296) 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.074+0000 7f8fa9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47306/0 (socket says 172.21.15.90:47306) 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac170890 con 0x7f8fac0cce50 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9400640 1 -- 172.21.15.90:0/292964491 learned_addr learned my addr 172.21.15.90:0/292964491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac170710 con 0x7f8fac06ed80 2024-03-31T15:54:17.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8fac170a10 con 0x7f8fac072c30 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8faa800640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fac0cce50 0x7f8fac164160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 -- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 msgr2=0x7f8fac163c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac163c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 -- 172.21.15.90:0/292964491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fac0cce50 msgr2=0x7f8fac164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fac0cce50 0x7f8fac164160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac1586b0 con 0x7f8fac06ed80 2024-03-31T15:54:17.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa9e00640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f940ab000 tx=0x7f8f9401ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940b3800 con 0x7f8fac06ed80 2024-03-31T15:54:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f940b39a0 con 0x7f8fac06ed80 2024-03-31T15:54:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac158990 con 0x7f8fac06ed80 2024-03-31T15:54:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f940b3dd0 con 0x7f8fac06ed80 2024-03-31T15:54:17.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.082+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac158f50 con 0x7f8fac06ed80 2024-03-31T15:54:17.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.086+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8f940b2380 con 0x7f8fac06ed80 2024-03-31T15:54:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.086+0000 7f8fa2000640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 0x7f8f8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.086+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8f94121350 con 0x7f8fac06ed80 2024-03-31T15:54:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.090+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fac068660 con 0x7f8fac06ed80 2024-03-31T15:54:17.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.090+0000 7f8fa9400640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 0x7f8f8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.090+0000 7f8fa9400640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 0x7f8f8806a840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0c6a80 tx=0x7f8fa4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.098+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8f940c3060 con 0x7f8fac06ed80 2024-03-31T15:54:17.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.142+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1a300cf260 con 0x7f1a18003a10 2024-03-31T15:54:17.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.142+0000 7f1a2cc00640 1 -- 172.21.15.90:0/2894336337 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a24019040 con 0x7f1a18003a10 2024-03-31T15:54:17.143 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:17.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.154+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 msgr2=0x7f1a1006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.154+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 0x7f1a1006a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006e90 tx=0x7f1a1c006de0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.154+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 msgr2=0x7f1a30159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.154+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400c190 tx=0x7f1a18005840 comp rx=0 tx=0).stop 2024-03-31T15:54:17.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 shutdown_connections 2024-03-31T15:54:17.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1a100683b0 0x7f1a1006a870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a300b3d10 0x7f1a300b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1a300c85d0 0x7f1a3015a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 --2- 172.21.15.90:0/2894336337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1a18003a10 0x7f1a30159ae0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 >> 172.21.15.90:0/2894336337 conn(0x7f1a3008bfd0 msgr2=0x7f1a300b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 shutdown_connections 2024-03-31T15:54:17.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.158+0000 7f1a37600640 1 -- 172.21.15.90:0/2894336337 wait complete. 2024-03-31T15:54:17.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.0 2024-03-31T15:54:17.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.222+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f121c15af40 con 0x7f121c0c38d0 2024-03-31T15:54:17.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.222+0000 7f1218c00640 1 -- 172.21.15.90:0/4201968772 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1214086020 con 0x7f121c0c38d0 2024-03-31T15:54:17.219 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.222 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 msgr2=0x7f11fc06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 0x7f11fc06a7e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f120c001c10 tx=0x7f120c000fb0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 msgr2=0x7f121c0b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 0x7f121c0b0840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f121c069d00 tx=0x7f121400c650 comp rx=0 tx=0).stop 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 shutdown_connections 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11fc068320 0x7f11fc06a7e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f121c0c4200 0x7f121c0b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f121c0c38d0 0x7f121c0b0840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 --2- 172.21.15.90:0/4201968772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12000051d0 0x7f121c2007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.226+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 >> 172.21.15.90:0/4201968772 conn(0x7f121c0b1880 msgr2=0x7f121c15cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.230+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 shutdown_connections 2024-03-31T15:54:17.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.230+0000 7f1223600640 1 -- 172.21.15.90:0/4201968772 wait complete. 2024-03-31T15:54:17.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.3 2024-03-31T15:54:17.258 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.298+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f99680b2b90 con 0x7f99680ca8c0 2024-03-31T15:54:17.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9965600640 1 -- 172.21.15.90:0/1221712074 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f99680cacd0 con 0x7f99680ca8c0 2024-03-31T15:54:17.296 INFO:teuthology.orchestra.run.smithi090.stdout:60129542154 2024-03-31T15:54:17.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 msgr2=0x7f994806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 0x7f994806a890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f99680c9f60 tx=0x7f995c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:17.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 msgr2=0x7f99681578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f99540b0040 tx=0x7f9954019910 comp rx=0 tx=0).stop 2024-03-31T15:54:17.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 shutdown_connections 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99480683d0 0x7f994806a890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99681511b0 0x7f9968156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99680ca8c0 0x7f99681578b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 --2- 172.21.15.90:0/1221712074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f996806d9f0 0x7f9968150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 >> 172.21.15.90:0/1221712074 conn(0x7f99680b1880 msgr2=0x7f99680b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 shutdown_connections 2024-03-31T15:54:17.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.302+0000 7f9970600640 1 -- 172.21.15.90:0/1221712074 wait complete. 2024-03-31T15:54:17.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.310+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f29b80b9280 con 0x7f29b80b53f0 2024-03-31T15:54:17.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.310+0000 7f29afe00640 1 -- 172.21.15.90:0/1886338909 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f29b81607e0 con 0x7f29b80b53f0 2024-03-31T15:54:17.304 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 msgr2=0x7f299406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 0x7f299406a890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f29b00012f0 tx=0x7f29b0009040 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 msgr2=0x7f29b81602a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c420 tx=0x7f29a800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 shutdown_connections 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f29940683d0 0x7f299406a890 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29b80d4ba0 0x7f29b8160be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29b80b8310 0x7f29b80d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 --2- 172.21.15.90:0/1886338909 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29b80b53f0 0x7f29b81602a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 >> 172.21.15.90:0/1886338909 conn(0x7f29b80d3f60 msgr2=0x7f29b80b9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 shutdown_connections 2024-03-31T15:54:17.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.314+0000 7f29bd400640 1 -- 172.21.15.90:0/1886338909 wait complete. 2024-03-31T15:54:17.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.318+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f62780c4fd0 con 0x7f6278072c30 2024-03-31T15:54:17.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.318+0000 7f626e000640 1 -- 172.21.15.90:0/983615849 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6270059ee0 con 0x7f6278072c30 2024-03-31T15:54:17.314 INFO:teuthology.orchestra.run.smithi090.stdout:55834574858 2024-03-31T15:54:17.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 msgr2=0x7f625406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 0x7f625406a910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62680097c0 tx=0x7f62680a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 msgr2=0x7f62780cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 0x7f62780cd3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f627000e450 tx=0x7f627000e910 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 shutdown_connections 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6254068450 0x7f625406a910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6278072c30 0x7f62780cd3b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6278071220 0x7f6278150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 --2- 172.21.15.90:0/983615849 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f627806ed80 0x7f62781513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 >> 172.21.15.90:0/983615849 conn(0x7f62780b1a50 msgr2=0x7f62780ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 shutdown_connections 2024-03-31T15:54:17.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.326+0000 7f627d200640 1 -- 172.21.15.90:0/983615849 wait complete. 2024-03-31T15:54:17.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.2 2024-03-31T15:54:17.325 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.1 2024-03-31T15:54:17.335 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.352 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-03-31T15:54:17.352 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f581006ec90 con 0x7f58101511b0 2024-03-31T15:54:17.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5807400640 1 -- 172.21.15.90:0/1967741104 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57fc056280 con 0x7f58101511b0 2024-03-31T15:54:17.357 INFO:teuthology.orchestra.run.smithi090.stdout:64424509451 2024-03-31T15:54:17.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 msgr2=0x7f57f006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 0x7f57f006a840 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f58000af000 tx=0x7f5800004740 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 msgr2=0x7f581015a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c3c0 tx=0x7f57fc00c880 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 shutdown_connections 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f57f0068380 0x7f57f006a840 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f581015a820 0x7f58100cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58101511b0 0x7f581015a0f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 --2- 172.21.15.90:0/1967741104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58100caa90 0x7f581014e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.362+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 >> 172.21.15.90:0/1967741104 conn(0x7f58100b1880 msgr2=0x7f58100b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.366+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 shutdown_connections 2024-03-31T15:54:17.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.366+0000 7f5816000640 1 -- 172.21.15.90:0/1967741104 wait complete. 2024-03-31T15:54:17.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.366+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f249c005590 con 0x7f24c415d760 2024-03-31T15:54:17.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.366+0000 7f24baa00640 1 -- 172.21.15.90:0/3780737355 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f24b4059e50 con 0x7f24c415d760 2024-03-31T15:54:17.363 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:17.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 msgr2=0x7f24a006a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 0x7f24a006a910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009a40 tx=0x7f24bc00b040 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 msgr2=0x7f24c41f8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 0x7f24c41f8140 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c3f0 tx=0x7f24b400c8b0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 shutdown_connections 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24a0068450 0x7f24a006a910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24c415d760 0x7f24c41f8140 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24c40cd6a0 0x7f24c40b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 --2- 172.21.15.90:0/3780737355 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24c40cc990 0x7f24c40b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 >> 172.21.15.90:0/3780737355 conn(0x7f24c40b16a0 msgr2=0x7f24c4072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 shutdown_connections 2024-03-31T15:54:17.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.370+0000 7f24b3e00640 1 -- 172.21.15.90:0/3780737355 wait complete. 2024-03-31T15:54:17.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.4 2024-03-31T15:54:17.382 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.385 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.6 2024-03-31T15:54:17.385 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.406+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8fac068660 con 0x7f8fac06ed80 2024-03-31T15:54:17.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.406+0000 7f8fa2000640 1 -- 172.21.15.90:0/292964491 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f940a7070 con 0x7f8fac06ed80 2024-03-31T15:54:17.402 INFO:teuthology.orchestra.run.smithi090.stdout:64424509450 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 msgr2=0x7f8f8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 0x7f8f8806a840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0c6a80 tx=0x7f8fa4009040 comp rx=0 tx=0).stop 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 msgr2=0x7f8fac152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f940ab000 tx=0x7f8f9401ad10 comp rx=0 tx=0).stop 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 shutdown_connections 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f88068380 0x7f8f8806a840 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8fac0cce50 0x7f8fac164160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8fac072c30 0x7f8fac163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 --2- 172.21.15.90:0/292964491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fac06ed80 0x7f8fac152380 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 >> 172.21.15.90:0/292964491 conn(0x7f8fac0b1a50 msgr2=0x7f8fac0b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 shutdown_connections 2024-03-31T15:54:17.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.410+0000 7f8fb1200640 1 -- 172.21.15.90:0/292964491 wait complete. 2024-03-31T15:54:17.423 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.5 2024-03-31T15:54:17.423 DEBUG:teuthology.parallel:result is None 2024-03-31T15:54:17.423 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-31T15:54:17.424 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:54:17.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:54:17.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:54:17.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 Processor -- start 2024-03-31T15:54:17.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 -- start start 2024-03-31T15:54:17.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f466415c050 0x7f466415fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46640d0100 con 0x7f466406fcc0 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46640cfb40 con 0x7f466415c430 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f466c000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f46640cfe20 con 0x7f466415c050 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f466415c050 0x7f466415fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f466415c050 0x7f466415fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51602/0 (socket says 172.21.15.90:51602) 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4668c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f466406fcc0 0x7f46641600f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34026/0 (socket says 172.21.15.90:34026) 2024-03-31T15:54:17.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 -- 172.21.15.90:0/111434496 learned_addr learned my addr 172.21.15.90:0/111434496 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 -- 172.21.15.90:0/111434496 >> v1:172.21.15.102:6789/0 conn(0x7f466415c430 legacy=0x7f466406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 -- 172.21.15.90:0/111434496 >> v1:172.21.15.90:6789/0 conn(0x7f466406fcc0 legacy=0x7f46641600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.614+0000 7f4669600640 1 -- 172.21.15.90:0/111434496 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466406b690 con 0x7f466415c050 2024-03-31T15:54:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f4669600640 1 --2- 172.21.15.90:0/111434496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466415fbb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4658009870 tx=0x7f465801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8694305cbbc8e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f4663e00640 1 -- 172.21.15.90:0/111434496 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46580aa680 con 0x7f466415c050 2024-03-31T15:54:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f4663e00640 1 -- 172.21.15.90:0/111434496 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46580af070 con 0x7f466415c050 2024-03-31T15:54:17.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f4663e00640 1 -- 172.21.15.90:0/111434496 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46580aaa40 con 0x7f466415c050 2024-03-31T15:54:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 -- 172.21.15.90:0/111434496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 msgr2=0x7f466415fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 --2- 172.21.15.90:0/111434496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466415fbb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4658009870 tx=0x7f465801c560 comp rx=0 tx=0).stop 2024-03-31T15:54:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 -- 172.21.15.90:0/111434496 shutdown_connections 2024-03-31T15:54:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 --2- 172.21.15.90:0/111434496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466415fbb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 -- 172.21.15.90:0/111434496 >> 172.21.15.90:0/111434496 conn(0x7f46640d05d0 msgr2=0x7f46640d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 -- 172.21.15.90:0/111434496 shutdown_connections 2024-03-31T15:54:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.618+0000 7f466c000640 1 -- 172.21.15.90:0/111434496 wait complete. 2024-03-31T15:54:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 Processor -- start 2024-03-31T15:54:17.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- start start 2024-03-31T15:54:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f466406fcc0 0x7f4664135990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f466415c430 0x7f46641370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f466416a0e0 con 0x7f466415c050 2024-03-31T15:54:17.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4664169f60 con 0x7f466415c430 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f466416a260 con 0x7f466406fcc0 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51612/0 (socket says 172.21.15.90:51612) 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 -- 172.21.15.90:0/1699587891 learned_addr learned my addr 172.21.15.90:0/1699587891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4669600640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f466406fcc0 0x7f4664135990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466a000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f466415c430 0x7f46641370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 -- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f466406fcc0 msgr2=0x7f4664135990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f466406fcc0 0x7f4664135990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 -- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f466415c430 msgr2=0x7f46641370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f466415c430 0x7f46641370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 -- 172.21.15.90:0/1699587891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4664139870 con 0x7f466415c050 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4668c00640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f465000c490 tx=0x7f465000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650018070 con 0x7f466415c050 2024-03-31T15:54:17.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4650007480 con 0x7f466415c050 2024-03-31T15:54:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650004a10 con 0x7f466415c050 2024-03-31T15:54:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4664136050 con 0x7f466415c050 2024-03-31T15:54:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4664136610 con 0x7f466415c050 2024-03-31T15:54:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.622+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4664068d00 con 0x7f466415c050 2024-03-31T15:54:17.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.626+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f465001b070 con 0x7f466415c050 2024-03-31T15:54:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.626+0000 7f4661600640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 0x7f464406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.626+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4650085f90 con 0x7f466415c050 2024-03-31T15:54:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.626+0000 7f4669600640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 0x7f464406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.626+0000 7f4669600640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 0x7f464406a840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46580ae000 tx=0x7f46580a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.630+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4650056170 con 0x7f466415c050 2024-03-31T15:54:17.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.786+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f466406dc50 con 0x7f4644068380 2024-03-31T15:54:17.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.794+0000 7f4661600640 1 -- 172.21.15.90:0/1699587891 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332537 (secure 0 0 0) 0x7f466406dc50 con 0x7f4644068380 2024-03-31T15:54:17.793 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:17.795 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 msgr2=0x7f464406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 0x7f464406a840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f46580ae000 tx=0x7f46580a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 msgr2=0x7f466414bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f465000c490 tx=0x7f465000c950 comp rx=0 tx=0).stop 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 shutdown_connections 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4644068380 0x7f464406a840 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f466415c430 0x7f46641370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f466415c050 0x7f466414bb80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 --2- 172.21.15.90:0/1699587891 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f466406fcc0 0x7f4664135990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:17.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 >> 172.21.15.90:0/1699587891 conn(0x7f46640d05d0 msgr2=0x7f4664150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:17.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 shutdown_connections 2024-03-31T15:54:17.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.798+0000 7f466c000640 1 -- 172.21.15.90:0/1699587891 wait complete. 2024-03-31T15:54:17.816 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-31T15:54:17.222591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219416,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755304,"statfs":{"total":773094113280,"available":772869431296,"internally_reserved":0,"allocated":3317760,"data_stored":2007160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003781"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359793+0000","last_change":"2024-03-31T15:53:46.757375+0000","last_active":"2024-03-31T15:53:53.359793+0000","last_peered":"2024-03-31T15:53:53.359793+0000","last_clean":"2024-03-31T15:53:53.359793+0000","last_became_active":"2024-03-31T15:53:46.757092+0000","last_became_peered":"2024-03-31T15:53:46.757092+0000","last_unstale":"2024-03-31T15:53:53.359793+0000","last_undegraded":"2024-03-31T15:53:53.359793+0000","last_fullsized":"2024-03-31T15:53:53.359793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248347+0000","last_change":"2024-03-31T15:53:47.766865+0000","last_active":"2024-03-31T15:53:53.248347+0000","last_peered":"2024-03-31T15:53:53.248347+0000","last_clean":"2024-03-31T15:53:53.248347+0000","last_became_active":"2024-03-31T15:53:47.766202+0000","last_became_peered":"2024-03-31T15:53:47.766202+0000","last_unstale":"2024-03-31T15:53:53.248347+0000","last_undegraded":"2024-03-31T15:53:53.248347+0000","last_fullsized":"2024-03-31T15:53:53.248347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:31.997737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247733+0000","last_change":"2024-03-31T15:53:51.804152+0000","last_active":"2024-03-31T15:53:53.247733+0000","last_peered":"2024-03-31T15:53:53.247733+0000","last_clean":"2024-03-31T15:53:53.247733+0000","last_became_active":"2024-03-31T15:53:51.802511+0000","last_became_peered":"2024-03-31T15:53:51.802511+0000","last_unstale":"2024-03-31T15:53:53.247733+0000","last_undegraded":"2024-03-31T15:53:53.247733+0000","last_fullsized":"2024-03-31T15:53:53.247733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:38.965494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248547+0000","last_change":"2024-03-31T15:53:46.763714+0000","last_active":"2024-03-31T15:53:53.248547+0000","last_peered":"2024-03-31T15:53:53.248547+0000","last_clean":"2024-03-31T15:53:53.248547+0000","last_became_active":"2024-03-31T15:53:46.763376+0000","last_became_peered":"2024-03-31T15:53:46.763376+0000","last_unstale":"2024-03-31T15:53:53.248547+0000","last_undegraded":"2024-03-31T15:53:53.248547+0000","last_fullsized":"2024-03-31T15:53:53.248547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359675+0000","last_change":"2024-03-31T15:53:47.771438+0000","last_active":"2024-03-31T15:53:53.359675+0000","last_peered":"2024-03-31T15:53:53.359675+0000","last_clean":"2024-03-31T15:53:53.359675+0000","last_became_active":"2024-03-31T15:53:47.771175+0000","last_became_peered":"2024-03-31T15:53:47.771175+0000","last_unstale":"2024-03-31T15:53:53.359675+0000","last_undegraded":"2024-03-31T15:53:53.359675+0000","last_fullsized":"2024-03-31T15:53:53.359675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792499+0000","last_change":"2024-03-31T15:53:51.797656+0000","last_active":"2024-03-31T15:53:52.792499+0000","last_peered":"2024-03-31T15:53:52.792499+0000","last_clean":"2024-03-31T15:53:52.792499+0000","last_became_active":"2024-03-31T15:53:51.797356+0000","last_became_peered":"2024-03-31T15:53:51.797356+0000","last_unstale":"2024-03-31T15:53:52.792499+0000","last_undegraded":"2024-03-31T15:53:52.792499+0000","last_fullsized":"2024-03-31T15:53:52.792499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:05.027492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792090+0000","last_change":"2024-03-31T15:53:46.764787+0000","last_active":"2024-03-31T15:53:52.792090+0000","last_peered":"2024-03-31T15:53:52.792090+0000","last_clean":"2024-03-31T15:53:52.792090+0000","last_became_active":"2024-03-31T15:53:46.764484+0000","last_became_peered":"2024-03-31T15:53:46.764484+0000","last_unstale":"2024-03-31T15:53:52.792090+0000","last_undegraded":"2024-03-31T15:53:52.792090+0000","last_fullsized":"2024-03-31T15:53:52.792090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:15.262251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789729+0000","last_change":"2024-03-31T15:53:47.773460+0000","last_active":"2024-03-31T15:53:52.789729+0000","last_peered":"2024-03-31T15:53:52.789729+0000","last_clean":"2024-03-31T15:53:52.789729+0000","last_became_active":"2024-03-31T15:53:47.773148+0000","last_became_peered":"2024-03-31T15:53:47.773148+0000","last_unstale":"2024-03-31T15:53:52.789729+0000","last_undegraded":"2024-03-31T15:53:52.789729+0000","last_fullsized":"2024-03-31T15:53:52.789729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.396848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793233+0000","last_change":"2024-03-31T15:53:51.799743+0000","last_active":"2024-03-31T15:53:52.793233+0000","last_peered":"2024-03-31T15:53:52.793233+0000","last_clean":"2024-03-31T15:53:52.793233+0000","last_became_active":"2024-03-31T15:53:51.799332+0000","last_became_peered":"2024-03-31T15:53:51.799332+0000","last_unstale":"2024-03-31T15:53:52.793233+0000","last_undegraded":"2024-03-31T15:53:52.793233+0000","last_fullsized":"2024-03-31T15:53:52.793233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:58.431101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791817+0000","last_change":"2024-03-31T15:53:46.765429+0000","last_active":"2024-03-31T15:53:52.791817+0000","last_peered":"2024-03-31T15:53:52.791817+0000","last_clean":"2024-03-31T15:53:52.791817+0000","last_became_active":"2024-03-31T15:53:46.765129+0000","last_became_peered":"2024-03-31T15:53:46.765129+0000","last_unstale":"2024-03-31T15:53:52.791817+0000","last_undegraded":"2024-03-31T15:53:52.791817+0000","last_fullsized":"2024-03-31T15:53:52.791817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:00:22.460897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790207+0000","last_change":"2024-03-31T15:53:47.770281+0000","last_active":"2024-03-31T15:53:52.790207+0000","last_peered":"2024-03-31T15:53:52.790207+0000","last_clean":"2024-03-31T15:53:52.790207+0000","last_became_active":"2024-03-31T15:53:47.769991+0000","last_became_peered":"2024-03-31T15:53:47.769991+0000","last_unstale":"2024-03-31T15:53:52.790207+0000","last_undegraded":"2024-03-31T15:53:52.790207+0000","last_fullsized":"2024-03-31T15:53:52.790207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:20.227687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792734+0000","last_change":"2024-03-31T15:53:51.797162+0000","last_active":"2024-03-31T15:53:52.792734+0000","last_peered":"2024-03-31T15:53:52.792734+0000","last_clean":"2024-03-31T15:53:52.792734+0000","last_became_active":"2024-03-31T15:53:51.796702+0000","last_became_peered":"2024-03-31T15:53:51.796702+0000","last_unstale":"2024-03-31T15:53:52.792734+0000","last_undegraded":"2024-03-31T15:53:52.792734+0000","last_fullsized":"2024-03-31T15:53:52.792734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:40.593467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306348+0000","last_change":"2024-03-31T15:53:47.762689+0000","last_active":"2024-03-31T15:53:53.306348+0000","last_peered":"2024-03-31T15:53:53.306348+0000","last_clean":"2024-03-31T15:53:53.306348+0000","last_became_active":"2024-03-31T15:53:47.762427+0000","last_became_peered":"2024-03-31T15:53:47.762427+0000","last_unstale":"2024-03-31T15:53:53.306348+0000","last_undegraded":"2024-03-31T15:53:53.306348+0000","last_fullsized":"2024-03-31T15:53:53.306348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791816+0000","last_change":"2024-03-31T15:53:46.757859+0000","last_active":"2024-03-31T15:53:52.791816+0000","last_peered":"2024-03-31T15:53:52.791816+0000","last_clean":"2024-03-31T15:53:52.791816+0000","last_became_active":"2024-03-31T15:53:46.757618+0000","last_became_peered":"2024-03-31T15:53:46.757618+0000","last_unstale":"2024-03-31T15:53:52.791816+0000","last_undegraded":"2024-03-31T15:53:52.791816+0000","last_fullsized":"2024-03-31T15:53:52.791816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:08:18.634688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793519+0000","last_change":"2024-03-31T15:53:51.797915+0000","last_active":"2024-03-31T15:53:52.793519+0000","last_peered":"2024-03-31T15:53:52.793519+0000","last_clean":"2024-03-31T15:53:52.793519+0000","last_became_active":"2024-03-31T15:53:51.797515+0000","last_became_peered":"2024-03-31T15:53:51.797515+0000","last_unstale":"2024-03-31T15:53:52.793519+0000","last_undegraded":"2024-03-31T15:53:52.793519+0000","last_fullsized":"2024-03-31T15:53:52.793519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247998+0000","last_change":"2024-03-31T15:53:47.765185+0000","last_active":"2024-03-31T15:53:53.247998+0000","last_peered":"2024-03-31T15:53:53.247998+0000","last_clean":"2024-03-31T15:53:53.247998+0000","last_became_active":"2024-03-31T15:53:47.764777+0000","last_became_peered":"2024-03-31T15:53:47.764777+0000","last_unstale":"2024-03-31T15:53:53.247998+0000","last_undegraded":"2024-03-31T15:53:53.247998+0000","last_fullsized":"2024-03-31T15:53:53.247998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:23.224004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920654+0000","last_change":"2024-03-31T15:53:46.762534+0000","last_active":"2024-03-31T15:53:52.920654+0000","last_peered":"2024-03-31T15:53:52.920654+0000","last_clean":"2024-03-31T15:53:52.920654+0000","last_became_active":"2024-03-31T15:53:46.762081+0000","last_became_peered":"2024-03-31T15:53:46.762081+0000","last_unstale":"2024-03-31T15:53:52.920654+0000","last_undegraded":"2024-03-31T15:53:52.920654+0000","last_fullsized":"2024-03-31T15:53:52.920654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793296+0000","last_change":"2024-03-31T15:53:51.802744+0000","last_active":"2024-03-31T15:53:52.793296+0000","last_peered":"2024-03-31T15:53:52.793296+0000","last_clean":"2024-03-31T15:53:52.793296+0000","last_became_active":"2024-03-31T15:53:51.802328+0000","last_became_peered":"2024-03-31T15:53:51.802328+0000","last_unstale":"2024-03-31T15:53:52.793296+0000","last_undegraded":"2024-03-31T15:53:52.793296+0000","last_fullsized":"2024-03-31T15:53:52.793296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:00.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247904+0000","last_change":"2024-03-31T15:53:47.751398+0000","last_active":"2024-03-31T15:53:53.247904+0000","last_peered":"2024-03-31T15:53:53.247904+0000","last_clean":"2024-03-31T15:53:53.247904+0000","last_became_active":"2024-03-31T15:53:47.750758+0000","last_became_peered":"2024-03-31T15:53:47.750758+0000","last_unstale":"2024-03-31T15:53:53.247904+0000","last_undegraded":"2024-03-31T15:53:53.247904+0000","last_fullsized":"2024-03-31T15:53:53.247904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:31.275051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920257+0000","last_change":"2024-03-31T15:53:46.755068+0000","last_active":"2024-03-31T15:53:52.920257+0000","last_peered":"2024-03-31T15:53:52.920257+0000","last_clean":"2024-03-31T15:53:52.920257+0000","last_became_active":"2024-03-31T15:53:46.754754+0000","last_became_peered":"2024-03-31T15:53:46.754754+0000","last_unstale":"2024-03-31T15:53:52.920257+0000","last_undegraded":"2024-03-31T15:53:52.920257+0000","last_fullsized":"2024-03-31T15:53:52.920257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921340+0000","last_change":"2024-03-31T15:53:51.803182+0000","last_active":"2024-03-31T15:53:52.921340+0000","last_peered":"2024-03-31T15:53:52.921340+0000","last_clean":"2024-03-31T15:53:52.921340+0000","last_became_active":"2024-03-31T15:53:51.802710+0000","last_became_peered":"2024-03-31T15:53:51.802710+0000","last_unstale":"2024-03-31T15:53:52.921340+0000","last_undegraded":"2024-03-31T15:53:52.921340+0000","last_fullsized":"2024-03-31T15:53:52.921340+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:40.165393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792965+0000","last_change":"2024-03-31T15:53:47.755358+0000","last_active":"2024-03-31T15:53:52.792965+0000","last_peered":"2024-03-31T15:53:52.792965+0000","last_clean":"2024-03-31T15:53:52.792965+0000","last_became_active":"2024-03-31T15:53:47.754487+0000","last_became_peered":"2024-03-31T15:53:47.754487+0000","last_unstale":"2024-03-31T15:53:52.792965+0000","last_undegraded":"2024-03-31T15:53:52.792965+0000","last_fullsized":"2024-03-31T15:53:52.792965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:53.949585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791156+0000","last_change":"2024-03-31T15:53:46.755464+0000","last_active":"2024-03-31T15:53:52.791156+0000","last_peered":"2024-03-31T15:53:52.791156+0000","last_clean":"2024-03-31T15:53:52.791156+0000","last_became_active":"2024-03-31T15:53:46.754626+0000","last_became_peered":"2024-03-31T15:53:46.754626+0000","last_unstale":"2024-03-31T15:53:52.791156+0000","last_undegraded":"2024-03-31T15:53:52.791156+0000","last_fullsized":"2024-03-31T15:53:52.791156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:18.769264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793394+0000","last_change":"2024-03-31T15:53:51.803474+0000","last_active":"2024-03-31T15:53:52.793394+0000","last_peered":"2024-03-31T15:53:52.793394+0000","last_clean":"2024-03-31T15:53:52.793394+0000","last_became_active":"2024-03-31T15:53:51.803014+0000","last_became_peered":"2024-03-31T15:53:51.803014+0000","last_unstale":"2024-03-31T15:53:52.793394+0000","last_undegraded":"2024-03-31T15:53:52.793394+0000","last_fullsized":"2024-03-31T15:53:52.793394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359741+0000","last_change":"2024-03-31T15:53:47.754929+0000","last_active":"2024-03-31T15:53:53.359741+0000","last_peered":"2024-03-31T15:53:53.359741+0000","last_clean":"2024-03-31T15:53:53.359741+0000","last_became_active":"2024-03-31T15:53:47.754625+0000","last_became_peered":"2024-03-31T15:53:47.754625+0000","last_unstale":"2024-03-31T15:53:53.359741+0000","last_undegraded":"2024-03-31T15:53:53.359741+0000","last_fullsized":"2024-03-31T15:53:53.359741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307339+0000","last_change":"2024-03-31T15:53:46.757784+0000","last_active":"2024-03-31T15:53:53.307339+0000","last_peered":"2024-03-31T15:53:53.307339+0000","last_clean":"2024-03-31T15:53:53.307339+0000","last_became_active":"2024-03-31T15:53:46.757342+0000","last_became_peered":"2024-03-31T15:53:46.757342+0000","last_unstale":"2024-03-31T15:53:53.307339+0000","last_undegraded":"2024-03-31T15:53:53.307339+0000","last_fullsized":"2024-03-31T15:53:53.307339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792725+0000","last_change":"2024-03-31T15:53:51.800310+0000","last_active":"2024-03-31T15:53:52.792725+0000","last_peered":"2024-03-31T15:53:52.792725+0000","last_clean":"2024-03-31T15:53:52.792725+0000","last_became_active":"2024-03-31T15:53:51.798223+0000","last_became_peered":"2024-03-31T15:53:51.798223+0000","last_unstale":"2024-03-31T15:53:52.792725+0000","last_undegraded":"2024-03-31T15:53:52.792725+0000","last_fullsized":"2024-03-31T15:53:52.792725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248692+0000","last_change":"2024-03-31T15:53:47.754983+0000","last_active":"2024-03-31T15:53:53.248692+0000","last_peered":"2024-03-31T15:53:53.248692+0000","last_clean":"2024-03-31T15:53:53.248692+0000","last_became_active":"2024-03-31T15:53:47.754734+0000","last_became_peered":"2024-03-31T15:53:47.754734+0000","last_unstale":"2024-03-31T15:53:53.248692+0000","last_undegraded":"2024-03-31T15:53:53.248692+0000","last_fullsized":"2024-03-31T15:53:53.248692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:31:24.931497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921179+0000","last_change":"2024-03-31T15:53:46.763360+0000","last_active":"2024-03-31T15:53:52.921179+0000","last_peered":"2024-03-31T15:53:52.921179+0000","last_clean":"2024-03-31T15:53:52.921179+0000","last_became_active":"2024-03-31T15:53:46.763005+0000","last_became_peered":"2024-03-31T15:53:46.763005+0000","last_unstale":"2024-03-31T15:53:52.921179+0000","last_undegraded":"2024-03-31T15:53:52.921179+0000","last_fullsized":"2024-03-31T15:53:52.921179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:30.435282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247663+0000","last_change":"2024-03-31T15:53:51.800117+0000","last_active":"2024-03-31T15:53:53.247663+0000","last_peered":"2024-03-31T15:53:53.247663+0000","last_clean":"2024-03-31T15:53:53.247663+0000","last_became_active":"2024-03-31T15:53:51.799217+0000","last_became_peered":"2024-03-31T15:53:51.799217+0000","last_unstale":"2024-03-31T15:53:53.247663+0000","last_undegraded":"2024-03-31T15:53:53.247663+0000","last_fullsized":"2024-03-31T15:53:53.247663+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:39.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920855+0000","last_change":"2024-03-31T15:53:47.754428+0000","last_active":"2024-03-31T15:53:52.920855+0000","last_peered":"2024-03-31T15:53:52.920855+0000","last_clean":"2024-03-31T15:53:52.920855+0000","last_became_active":"2024-03-31T15:53:47.753085+0000","last_became_peered":"2024-03-31T15:53:47.753085+0000","last_unstale":"2024-03-31T15:53:52.920855+0000","last_undegraded":"2024-03-31T15:53:52.920855+0000","last_fullsized":"2024-03-31T15:53:52.920855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792251+0000","last_change":"2024-03-31T15:53:46.763802+0000","last_active":"2024-03-31T15:53:52.792251+0000","last_peered":"2024-03-31T15:53:52.792251+0000","last_clean":"2024-03-31T15:53:52.792251+0000","last_became_active":"2024-03-31T15:53:46.762025+0000","last_became_peered":"2024-03-31T15:53:46.762025+0000","last_unstale":"2024-03-31T15:53:52.792251+0000","last_undegraded":"2024-03-31T15:53:52.792251+0000","last_fullsized":"2024-03-31T15:53:52.792251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:50.235479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359173+0000","last_change":"2024-03-31T15:53:51.796387+0000","last_active":"2024-03-31T15:53:53.359173+0000","last_peered":"2024-03-31T15:53:53.359173+0000","last_clean":"2024-03-31T15:53:53.359173+0000","last_became_active":"2024-03-31T15:53:51.795890+0000","last_became_peered":"2024-03-31T15:53:51.795890+0000","last_unstale":"2024-03-31T15:53:53.359173+0000","last_undegraded":"2024-03-31T15:53:53.359173+0000","last_fullsized":"2024-03-31T15:53:53.359173+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790297+0000","last_change":"2024-03-31T15:53:47.765672+0000","last_active":"2024-03-31T15:53:52.790297+0000","last_peered":"2024-03-31T15:53:52.790297+0000","last_clean":"2024-03-31T15:53:52.790297+0000","last_became_active":"2024-03-31T15:53:47.765481+0000","last_became_peered":"2024-03-31T15:53:47.765481+0000","last_unstale":"2024-03-31T15:53:52.790297+0000","last_undegraded":"2024-03-31T15:53:52.790297+0000","last_fullsized":"2024-03-31T15:53:52.790297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:14.332486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920943+0000","last_change":"2024-03-31T15:53:46.752123+0000","last_active":"2024-03-31T15:53:52.920943+0000","last_peered":"2024-03-31T15:53:52.920943+0000","last_clean":"2024-03-31T15:53:52.920943+0000","last_became_active":"2024-03-31T15:53:46.751752+0000","last_became_peered":"2024-03-31T15:53:46.751752+0000","last_unstale":"2024-03-31T15:53:52.920943+0000","last_undegraded":"2024-03-31T15:53:52.920943+0000","last_fullsized":"2024-03-31T15:53:52.920943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920358+0000","last_change":"2024-03-31T15:53:51.798289+0000","last_active":"2024-03-31T15:53:52.920358+0000","last_peered":"2024-03-31T15:53:52.920358+0000","last_clean":"2024-03-31T15:53:52.920358+0000","last_became_active":"2024-03-31T15:53:51.797843+0000","last_became_peered":"2024-03-31T15:53:51.797843+0000","last_unstale":"2024-03-31T15:53:52.920358+0000","last_undegraded":"2024-03-31T15:53:52.920358+0000","last_fullsized":"2024-03-31T15:53:52.920358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793832+0000","last_change":"2024-03-31T15:53:47.746865+0000","last_active":"2024-03-31T15:53:52.793832+0000","last_peered":"2024-03-31T15:53:52.793832+0000","last_clean":"2024-03-31T15:53:52.793832+0000","last_became_active":"2024-03-31T15:53:47.746098+0000","last_became_peered":"2024-03-31T15:53:47.746098+0000","last_unstale":"2024-03-31T15:53:52.793832+0000","last_undegraded":"2024-03-31T15:53:52.793832+0000","last_fullsized":"2024-03-31T15:53:52.793832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:31.443157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790702+0000","last_change":"2024-03-31T15:53:46.753263+0000","last_active":"2024-03-31T15:53:52.790702+0000","last_peered":"2024-03-31T15:53:52.790702+0000","last_clean":"2024-03-31T15:53:52.790702+0000","last_became_active":"2024-03-31T15:53:46.751947+0000","last_became_peered":"2024-03-31T15:53:46.751947+0000","last_unstale":"2024-03-31T15:53:52.790702+0000","last_undegraded":"2024-03-31T15:53:52.790702+0000","last_fullsized":"2024-03-31T15:53:52.790702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920882+0000","last_change":"2024-03-31T15:53:51.797078+0000","last_active":"2024-03-31T15:53:52.920882+0000","last_peered":"2024-03-31T15:53:52.920882+0000","last_clean":"2024-03-31T15:53:52.920882+0000","last_became_active":"2024-03-31T15:53:51.796451+0000","last_became_peered":"2024-03-31T15:53:51.796451+0000","last_unstale":"2024-03-31T15:53:52.920882+0000","last_undegraded":"2024-03-31T15:53:52.920882+0000","last_fullsized":"2024-03-31T15:53:52.920882+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920377+0000","last_change":"2024-03-31T15:53:47.752652+0000","last_active":"2024-03-31T15:53:52.920377+0000","last_peered":"2024-03-31T15:53:52.920377+0000","last_clean":"2024-03-31T15:53:52.920377+0000","last_became_active":"2024-03-31T15:53:47.752105+0000","last_became_peered":"2024-03-31T15:53:47.752105+0000","last_unstale":"2024-03-31T15:53:52.920377+0000","last_undegraded":"2024-03-31T15:53:52.920377+0000","last_fullsized":"2024-03-31T15:53:52.920377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528393+0000","last_change":"2024-03-31T15:53:46.753745+0000","last_active":"2024-03-31T15:53:53.528393+0000","last_peered":"2024-03-31T15:53:53.528393+0000","last_clean":"2024-03-31T15:53:53.528393+0000","last_became_active":"2024-03-31T15:53:46.753423+0000","last_became_peered":"2024-03-31T15:53:46.753423+0000","last_unstale":"2024-03-31T15:53:53.528393+0000","last_undegraded":"2024-03-31T15:53:53.528393+0000","last_fullsized":"2024-03-31T15:53:53.528393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921329+0000","last_change":"2024-03-31T15:53:51.802368+0000","last_active":"2024-03-31T15:53:52.921329+0000","last_peered":"2024-03-31T15:53:52.921329+0000","last_clean":"2024-03-31T15:53:52.921329+0000","last_became_active":"2024-03-31T15:53:51.801946+0000","last_became_peered":"2024-03-31T15:53:51.801946+0000","last_unstale":"2024-03-31T15:53:52.921329+0000","last_undegraded":"2024-03-31T15:53:52.921329+0000","last_fullsized":"2024-03-31T15:53:52.921329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.358775+0000","last_change":"2024-03-31T15:53:47.745539+0000","last_active":"2024-03-31T15:53:53.358775+0000","last_peered":"2024-03-31T15:53:53.358775+0000","last_clean":"2024-03-31T15:53:53.358775+0000","last_became_active":"2024-03-31T15:53:47.745265+0000","last_became_peered":"2024-03-31T15:53:47.745265+0000","last_unstale":"2024-03-31T15:53:53.358775+0000","last_undegraded":"2024-03-31T15:53:53.358775+0000","last_fullsized":"2024-03-31T15:53:53.358775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:12:22.550188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791245+0000","last_change":"2024-03-31T15:53:46.761890+0000","last_active":"2024-03-31T15:53:52.791245+0000","last_peered":"2024-03-31T15:53:52.791245+0000","last_clean":"2024-03-31T15:53:52.791245+0000","last_became_active":"2024-03-31T15:53:46.761417+0000","last_became_peered":"2024-03-31T15:53:46.761417+0000","last_unstale":"2024-03-31T15:53:52.791245+0000","last_undegraded":"2024-03-31T15:53:52.791245+0000","last_fullsized":"2024-03-31T15:53:52.791245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:25.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359685+0000","last_change":"2024-03-31T15:53:51.796610+0000","last_active":"2024-03-31T15:53:53.359685+0000","last_peered":"2024-03-31T15:53:53.359685+0000","last_clean":"2024-03-31T15:53:53.359685+0000","last_became_active":"2024-03-31T15:53:51.796112+0000","last_became_peered":"2024-03-31T15:53:51.796112+0000","last_unstale":"2024-03-31T15:53:53.359685+0000","last_undegraded":"2024-03-31T15:53:53.359685+0000","last_fullsized":"2024-03-31T15:53:53.359685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247912+0000","last_change":"2024-03-31T15:53:47.768681+0000","last_active":"2024-03-31T15:53:53.247912+0000","last_peered":"2024-03-31T15:53:53.247912+0000","last_clean":"2024-03-31T15:53:53.247912+0000","last_became_active":"2024-03-31T15:53:47.768416+0000","last_became_peered":"2024-03-31T15:53:47.768416+0000","last_unstale":"2024-03-31T15:53:53.247912+0000","last_undegraded":"2024-03-31T15:53:53.247912+0000","last_fullsized":"2024-03-31T15:53:53.247912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:02.383327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789855+0000","last_change":"2024-03-31T15:53:46.760553+0000","last_active":"2024-03-31T15:53:52.789855+0000","last_peered":"2024-03-31T15:53:52.789855+0000","last_clean":"2024-03-31T15:53:52.789855+0000","last_became_active":"2024-03-31T15:53:46.760258+0000","last_became_peered":"2024-03-31T15:53:46.760258+0000","last_unstale":"2024-03-31T15:53:52.789855+0000","last_undegraded":"2024-03-31T15:53:52.789855+0000","last_fullsized":"2024-03-31T15:53:52.789855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791305+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:53:52.791305+0000","last_peered":"2024-03-31T15:53:52.791305+0000","last_clean":"2024-03-31T15:53:52.791305+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:53:52.791305+0000","last_undegraded":"2024-03-31T15:53:52.791305+0000","last_fullsized":"2024-03-31T15:53:52.791305+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306820+0000","last_change":"2024-03-31T15:53:51.789901+0000","last_active":"2024-03-31T15:53:53.306820+0000","last_peered":"2024-03-31T15:53:53.306820+0000","last_clean":"2024-03-31T15:53:53.306820+0000","last_became_active":"2024-03-31T15:53:51.789529+0000","last_became_peered":"2024-03-31T15:53:51.789529+0000","last_unstale":"2024-03-31T15:53:53.306820+0000","last_undegraded":"2024-03-31T15:53:53.306820+0000","last_fullsized":"2024-03-31T15:53:53.306820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791956+0000","last_change":"2024-03-31T15:53:47.764710+0000","last_active":"2024-03-31T15:53:52.791956+0000","last_peered":"2024-03-31T15:53:52.791956+0000","last_clean":"2024-03-31T15:53:52.791956+0000","last_became_active":"2024-03-31T15:53:47.764408+0000","last_became_peered":"2024-03-31T15:53:47.764408+0000","last_unstale":"2024-03-31T15:53:52.791956+0000","last_undegraded":"2024-03-31T15:53:52.791956+0000","last_fullsized":"2024-03-31T15:53:52.791956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:14:45.512051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528719+0000","last_change":"2024-03-31T15:53:46.757343+0000","last_active":"2024-03-31T15:53:53.528719+0000","last_peered":"2024-03-31T15:53:53.528719+0000","last_clean":"2024-03-31T15:53:53.528719+0000","last_became_active":"2024-03-31T15:53:46.757028+0000","last_became_peered":"2024-03-31T15:53:46.757028+0000","last_unstale":"2024-03-31T15:53:53.528719+0000","last_undegraded":"2024-03-31T15:53:53.528719+0000","last_fullsized":"2024-03-31T15:53:53.528719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:19.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306461+0000","last_change":"2024-03-31T15:53:51.799696+0000","last_active":"2024-03-31T15:53:53.306461+0000","last_peered":"2024-03-31T15:53:53.306461+0000","last_clean":"2024-03-31T15:53:53.306461+0000","last_became_active":"2024-03-31T15:53:51.799041+0000","last_became_peered":"2024-03-31T15:53:51.799041+0000","last_unstale":"2024-03-31T15:53:53.306461+0000","last_undegraded":"2024-03-31T15:53:53.306461+0000","last_fullsized":"2024-03-31T15:53:53.306461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789786+0000","last_change":"2024-03-31T15:53:47.767189+0000","last_active":"2024-03-31T15:53:52.789786+0000","last_peered":"2024-03-31T15:53:52.789786+0000","last_clean":"2024-03-31T15:53:52.789786+0000","last_became_active":"2024-03-31T15:53:47.766941+0000","last_became_peered":"2024-03-31T15:53:47.766941+0000","last_unstale":"2024-03-31T15:53:52.789786+0000","last_undegraded":"2024-03-31T15:53:52.789786+0000","last_fullsized":"2024-03-31T15:53:52.789786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:56.064763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921010+0000","last_change":"2024-03-31T15:53:46.755784+0000","last_active":"2024-03-31T15:53:52.921010+0000","last_peered":"2024-03-31T15:53:52.921010+0000","last_clean":"2024-03-31T15:53:52.921010+0000","last_became_active":"2024-03-31T15:53:46.755518+0000","last_became_peered":"2024-03-31T15:53:46.755518+0000","last_unstale":"2024-03-31T15:53:52.921010+0000","last_undegraded":"2024-03-31T15:53:52.921010+0000","last_fullsized":"2024-03-31T15:53:52.921010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:03.068278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359266+0000","last_change":"2024-03-31T15:53:51.805030+0000","last_active":"2024-03-31T15:53:53.359266+0000","last_peered":"2024-03-31T15:53:53.359266+0000","last_clean":"2024-03-31T15:53:53.359266+0000","last_became_active":"2024-03-31T15:53:51.804608+0000","last_became_peered":"2024-03-31T15:53:51.804608+0000","last_unstale":"2024-03-31T15:53:53.359266+0000","last_undegraded":"2024-03-31T15:53:53.359266+0000","last_fullsized":"2024-03-31T15:53:53.359266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792768+0000","last_change":"2024-03-31T15:53:47.763067+0000","last_active":"2024-03-31T15:53:52.792768+0000","last_peered":"2024-03-31T15:53:52.792768+0000","last_clean":"2024-03-31T15:53:52.792768+0000","last_became_active":"2024-03-31T15:53:47.762801+0000","last_became_peered":"2024-03-31T15:53:47.762801+0000","last_unstale":"2024-03-31T15:53:52.792768+0000","last_undegraded":"2024-03-31T15:53:52.792768+0000","last_fullsized":"2024-03-31T15:53:52.792768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:34.931240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528883+0000","last_change":"2024-03-31T15:53:46.763307+0000","last_active":"2024-03-31T15:53:53.528883+0000","last_peered":"2024-03-31T15:53:53.528883+0000","last_clean":"2024-03-31T15:53:53.528883+0000","last_became_active":"2024-03-31T15:53:46.763083+0000","last_became_peered":"2024-03-31T15:53:46.763083+0000","last_unstale":"2024-03-31T15:53:53.528883+0000","last_undegraded":"2024-03-31T15:53:53.528883+0000","last_fullsized":"2024-03-31T15:53:53.528883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:35:35.431822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789351+0000","last_change":"2024-03-31T15:53:51.792009+0000","last_active":"2024-03-31T15:53:52.789351+0000","last_peered":"2024-03-31T15:53:52.789351+0000","last_clean":"2024-03-31T15:53:52.789351+0000","last_became_active":"2024-03-31T15:53:51.791660+0000","last_became_peered":"2024-03-31T15:53:51.791660+0000","last_unstale":"2024-03-31T15:53:52.789351+0000","last_undegraded":"2024-03-31T15:53:52.789351+0000","last_fullsized":"2024-03-31T15:53:52.789351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:17.783876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792256+0000","last_change":"2024-03-31T15:53:47.771025+0000","last_active":"2024-03-31T15:53:52.792256+0000","last_peered":"2024-03-31T15:53:52.792256+0000","last_clean":"2024-03-31T15:53:52.792256+0000","last_became_active":"2024-03-31T15:53:47.769254+0000","last_became_peered":"2024-03-31T15:53:47.769254+0000","last_unstale":"2024-03-31T15:53:52.792256+0000","last_undegraded":"2024-03-31T15:53:52.792256+0000","last_fullsized":"2024-03-31T15:53:52.792256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:01.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793608+0000","last_change":"2024-03-31T15:53:46.759316+0000","last_active":"2024-03-31T15:53:52.793608+0000","last_peered":"2024-03-31T15:53:52.793608+0000","last_clean":"2024-03-31T15:53:52.793608+0000","last_became_active":"2024-03-31T15:53:46.758973+0000","last_became_peered":"2024-03-31T15:53:46.758973+0000","last_unstale":"2024-03-31T15:53:52.793608+0000","last_undegraded":"2024-03-31T15:53:52.793608+0000","last_fullsized":"2024-03-31T15:53:52.793608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:39.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792641+0000","last_change":"2024-03-31T15:53:51.795201+0000","last_active":"2024-03-31T15:53:52.792641+0000","last_peered":"2024-03-31T15:53:52.792641+0000","last_clean":"2024-03-31T15:53:52.792641+0000","last_became_active":"2024-03-31T15:53:51.794775+0000","last_became_peered":"2024-03-31T15:53:51.794775+0000","last_unstale":"2024-03-31T15:53:52.792641+0000","last_undegraded":"2024-03-31T15:53:52.792641+0000","last_fullsized":"2024-03-31T15:53:52.792641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:11.574086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791547+0000","last_change":"2024-03-31T15:53:47.760294+0000","last_active":"2024-03-31T15:53:52.791547+0000","last_peered":"2024-03-31T15:53:52.791547+0000","last_clean":"2024-03-31T15:53:52.791547+0000","last_became_active":"2024-03-31T15:53:47.760011+0000","last_became_peered":"2024-03-31T15:53:47.760011+0000","last_unstale":"2024-03-31T15:53:52.791547+0000","last_undegraded":"2024-03-31T15:53:52.791547+0000","last_fullsized":"2024-03-31T15:53:52.791547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:11.819602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528298+0000","last_change":"2024-03-31T15:53:46.759221+0000","last_active":"2024-03-31T15:53:53.528298+0000","last_peered":"2024-03-31T15:53:53.528298+0000","last_clean":"2024-03-31T15:53:53.528298+0000","last_became_active":"2024-03-31T15:53:46.758926+0000","last_became_peered":"2024-03-31T15:53:46.758926+0000","last_unstale":"2024-03-31T15:53:53.528298+0000","last_undegraded":"2024-03-31T15:53:53.528298+0000","last_fullsized":"2024-03-31T15:53:53.528298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359571+0000","last_change":"2024-03-31T15:53:51.794069+0000","last_active":"2024-03-31T15:53:53.359571+0000","last_peered":"2024-03-31T15:53:53.359571+0000","last_clean":"2024-03-31T15:53:53.359571+0000","last_became_active":"2024-03-31T15:53:51.793665+0000","last_became_peered":"2024-03-31T15:53:51.793665+0000","last_unstale":"2024-03-31T15:53:53.359571+0000","last_undegraded":"2024-03-31T15:53:53.359571+0000","last_fullsized":"2024-03-31T15:53:53.359571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306482+0000","last_change":"2024-03-31T15:53:47.763992+0000","last_active":"2024-03-31T15:53:53.306482+0000","last_peered":"2024-03-31T15:53:53.306482+0000","last_clean":"2024-03-31T15:53:53.306482+0000","last_became_active":"2024-03-31T15:53:47.763743+0000","last_became_peered":"2024-03-31T15:53:47.763743+0000","last_unstale":"2024-03-31T15:53:53.306482+0000","last_undegraded":"2024-03-31T15:53:53.306482+0000","last_fullsized":"2024-03-31T15:53:53.306482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306483+0000","last_change":"2024-03-31T15:53:46.764342+0000","last_active":"2024-03-31T15:53:53.306483+0000","last_peered":"2024-03-31T15:53:53.306483+0000","last_clean":"2024-03-31T15:53:53.306483+0000","last_became_active":"2024-03-31T15:53:46.763064+0000","last_became_peered":"2024-03-31T15:53:46.763064+0000","last_unstale":"2024-03-31T15:53:53.306483+0000","last_undegraded":"2024-03-31T15:53:53.306483+0000","last_fullsized":"2024-03-31T15:53:53.306483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248662+0000","last_change":"2024-03-31T15:53:51.802982+0000","last_active":"2024-03-31T15:53:53.248662+0000","last_peered":"2024-03-31T15:53:53.248662+0000","last_clean":"2024-03-31T15:53:53.248662+0000","last_became_active":"2024-03-31T15:53:51.802619+0000","last_became_peered":"2024-03-31T15:53:51.802619+0000","last_unstale":"2024-03-31T15:53:53.248662+0000","last_undegraded":"2024-03-31T15:53:53.248662+0000","last_fullsized":"2024-03-31T15:53:53.248662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:12:18.187743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791793+0000","last_change":"2024-03-31T15:53:47.763046+0000","last_active":"2024-03-31T15:53:52.791793+0000","last_peered":"2024-03-31T15:53:52.791793+0000","last_clean":"2024-03-31T15:53:52.791793+0000","last_became_active":"2024-03-31T15:53:47.762791+0000","last_became_peered":"2024-03-31T15:53:47.762791+0000","last_unstale":"2024-03-31T15:53:52.791793+0000","last_undegraded":"2024-03-31T15:53:52.791793+0000","last_fullsized":"2024-03-31T15:53:52.791793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:32:01.910382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790646+0000","last_change":"2024-03-31T15:53:46.765124+0000","last_active":"2024-03-31T15:53:52.790646+0000","last_peered":"2024-03-31T15:53:52.790646+0000","last_clean":"2024-03-31T15:53:52.790646+0000","last_became_active":"2024-03-31T15:53:46.764856+0000","last_became_peered":"2024-03-31T15:53:46.764856+0000","last_unstale":"2024-03-31T15:53:52.790646+0000","last_undegraded":"2024-03-31T15:53:52.790646+0000","last_fullsized":"2024-03-31T15:53:52.790646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248292+0000","last_change":"2024-03-31T15:53:51.807271+0000","last_active":"2024-03-31T15:53:53.248292+0000","last_peered":"2024-03-31T15:53:53.248292+0000","last_clean":"2024-03-31T15:53:53.248292+0000","last_became_active":"2024-03-31T15:53:51.806909+0000","last_became_peered":"2024-03-31T15:53:51.806909+0000","last_unstale":"2024-03-31T15:53:53.248292+0000","last_undegraded":"2024-03-31T15:53:53.248292+0000","last_fullsized":"2024-03-31T15:53:53.248292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:14:12.015472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359081+0000","last_change":"2024-03-31T15:53:47.762410+0000","last_active":"2024-03-31T15:53:53.359081+0000","last_peered":"2024-03-31T15:53:53.359081+0000","last_clean":"2024-03-31T15:53:53.359081+0000","last_became_active":"2024-03-31T15:53:47.762145+0000","last_became_peered":"2024-03-31T15:53:47.762145+0000","last_unstale":"2024-03-31T15:53:53.359081+0000","last_undegraded":"2024-03-31T15:53:53.359081+0000","last_fullsized":"2024-03-31T15:53:53.359081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359350+0000","last_change":"2024-03-31T15:53:46.756308+0000","last_active":"2024-03-31T15:53:53.359350+0000","last_peered":"2024-03-31T15:53:53.359350+0000","last_clean":"2024-03-31T15:53:53.359350+0000","last_became_active":"2024-03-31T15:53:46.756010+0000","last_became_peered":"2024-03-31T15:53:46.756010+0000","last_unstale":"2024-03-31T15:53:53.359350+0000","last_undegraded":"2024-03-31T15:53:53.359350+0000","last_fullsized":"2024-03-31T15:53:53.359350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:30:06.680148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790084+0000","last_change":"2024-03-31T15:53:51.806516+0000","last_active":"2024-03-31T15:53:52.790084+0000","last_peered":"2024-03-31T15:53:52.790084+0000","last_clean":"2024-03-31T15:53:52.790084+0000","last_became_active":"2024-03-31T15:53:51.805985+0000","last_became_peered":"2024-03-31T15:53:51.805985+0000","last_unstale":"2024-03-31T15:53:52.790084+0000","last_undegraded":"2024-03-31T15:53:52.790084+0000","last_fullsized":"2024-03-31T15:53:52.790084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:12:27.917062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791963+0000","last_change":"2024-03-31T15:53:47.768000+0000","last_active":"2024-03-31T15:53:52.791963+0000","last_peered":"2024-03-31T15:53:52.791963+0000","last_clean":"2024-03-31T15:53:52.791963+0000","last_became_active":"2024-03-31T15:53:47.767710+0000","last_became_peered":"2024-03-31T15:53:47.767710+0000","last_unstale":"2024-03-31T15:53:52.791963+0000","last_undegraded":"2024-03-31T15:53:52.791963+0000","last_fullsized":"2024-03-31T15:53:52.791963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:35.607484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:54:02.857882+0000","last_change":"2024-03-31T15:53:46.764364+0000","last_active":"2024-03-31T15:54:02.857882+0000","last_peered":"2024-03-31T15:54:02.857882+0000","last_clean":"2024-03-31T15:54:02.857882+0000","last_became_active":"2024-03-31T15:53:46.763234+0000","last_became_peered":"2024-03-31T15:53:46.763234+0000","last_unstale":"2024-03-31T15:54:02.857882+0000","last_undegraded":"2024-03-31T15:54:02.857882+0000","last_fullsized":"2024-03-31T15:54:02.857882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920563+0000","last_change":"2024-03-31T15:53:51.794393+0000","last_active":"2024-03-31T15:53:52.920563+0000","last_peered":"2024-03-31T15:53:52.920563+0000","last_clean":"2024-03-31T15:53:52.920563+0000","last_became_active":"2024-03-31T15:53:51.793984+0000","last_became_peered":"2024-03-31T15:53:51.793984+0000","last_unstale":"2024-03-31T15:53:52.920563+0000","last_undegraded":"2024-03-31T15:53:52.920563+0000","last_fullsized":"2024-03-31T15:53:52.920563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793726+0000","last_change":"2024-03-31T15:53:47.759150+0000","last_active":"2024-03-31T15:53:52.793726+0000","last_peered":"2024-03-31T15:53:52.793726+0000","last_clean":"2024-03-31T15:53:52.793726+0000","last_became_active":"2024-03-31T15:53:47.757846+0000","last_became_peered":"2024-03-31T15:53:47.757846+0000","last_unstale":"2024-03-31T15:53:52.793726+0000","last_undegraded":"2024-03-31T15:53:52.793726+0000","last_fullsized":"2024-03-31T15:53:52.793726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:16.643130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920932+0000","last_change":"2024-03-31T15:53:46.763645+0000","last_active":"2024-03-31T15:53:52.920932+0000","last_peered":"2024-03-31T15:53:52.920932+0000","last_clean":"2024-03-31T15:53:52.920932+0000","last_became_active":"2024-03-31T15:53:46.763370+0000","last_became_peered":"2024-03-31T15:53:46.763370+0000","last_unstale":"2024-03-31T15:53:52.920932+0000","last_undegraded":"2024-03-31T15:53:52.920932+0000","last_fullsized":"2024-03-31T15:53:52.920932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:15.635255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793545+0000","last_change":"2024-03-31T15:53:51.804406+0000","last_active":"2024-03-31T15:53:52.793545+0000","last_peered":"2024-03-31T15:53:52.793545+0000","last_clean":"2024-03-31T15:53:52.793545+0000","last_became_active":"2024-03-31T15:53:51.803887+0000","last_became_peered":"2024-03-31T15:53:51.803887+0000","last_unstale":"2024-03-31T15:53:52.793545+0000","last_undegraded":"2024-03-31T15:53:52.793545+0000","last_fullsized":"2024-03-31T15:53:52.793545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359778+0000","last_change":"2024-03-31T15:53:47.771467+0000","last_active":"2024-03-31T15:53:53.359778+0000","last_peered":"2024-03-31T15:53:53.359778+0000","last_clean":"2024-03-31T15:53:53.359778+0000","last_became_active":"2024-03-31T15:53:47.771267+0000","last_became_peered":"2024-03-31T15:53:47.771267+0000","last_unstale":"2024-03-31T15:53:53.359778+0000","last_undegraded":"2024-03-31T15:53:53.359778+0000","last_fullsized":"2024-03-31T15:53:53.359778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790236+0000","last_change":"2024-03-31T15:53:46.758836+0000","last_active":"2024-03-31T15:53:52.790236+0000","last_peered":"2024-03-31T15:53:52.790236+0000","last_clean":"2024-03-31T15:53:52.790236+0000","last_became_active":"2024-03-31T15:53:46.758359+0000","last_became_peered":"2024-03-31T15:53:46.758359+0000","last_unstale":"2024-03-31T15:53:52.790236+0000","last_undegraded":"2024-03-31T15:53:52.790236+0000","last_fullsized":"2024-03-31T15:53:52.790236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247843+0000","last_change":"2024-03-31T15:53:51.800437+0000","last_active":"2024-03-31T15:53:53.247843+0000","last_peered":"2024-03-31T15:53:53.247843+0000","last_clean":"2024-03-31T15:53:53.247843+0000","last_became_active":"2024-03-31T15:53:51.799728+0000","last_became_peered":"2024-03-31T15:53:51.799728+0000","last_unstale":"2024-03-31T15:53:53.247843+0000","last_undegraded":"2024-03-31T15:53:53.247843+0000","last_fullsized":"2024-03-31T15:53:53.247843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:45.272362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790191+0000","last_change":"2024-03-31T15:53:47.765037+0000","last_active":"2024-03-31T15:53:52.790191+0000","last_peered":"2024-03-31T15:53:52.790191+0000","last_clean":"2024-03-31T15:53:52.790191+0000","last_became_active":"2024-03-31T15:53:47.764764+0000","last_became_peered":"2024-03-31T15:53:47.764764+0000","last_unstale":"2024-03-31T15:53:52.790191+0000","last_undegraded":"2024-03-31T15:53:52.790191+0000","last_fullsized":"2024-03-31T15:53:52.790191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:07.287524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307175+0000","last_change":"2024-03-31T15:53:46.765922+0000","last_active":"2024-03-31T15:53:53.307175+0000","last_peered":"2024-03-31T15:53:53.307175+0000","last_clean":"2024-03-31T15:53:53.307175+0000","last_became_active":"2024-03-31T15:53:46.765530+0000","last_became_peered":"2024-03-31T15:53:46.765530+0000","last_unstale":"2024-03-31T15:53:53.307175+0000","last_undegraded":"2024-03-31T15:53:53.307175+0000","last_fullsized":"2024-03-31T15:53:53.307175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247784+0000","last_change":"2024-03-31T15:53:51.803591+0000","last_active":"2024-03-31T15:53:53.247784+0000","last_peered":"2024-03-31T15:53:53.247784+0000","last_clean":"2024-03-31T15:53:53.247784+0000","last_became_active":"2024-03-31T15:53:51.803139+0000","last_became_peered":"2024-03-31T15:53:51.803139+0000","last_unstale":"2024-03-31T15:53:53.247784+0000","last_undegraded":"2024-03-31T15:53:53.247784+0000","last_fullsized":"2024-03-31T15:53:53.247784+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:08.680601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359558+0000","last_change":"2024-03-31T15:53:51.809399+0000","last_active":"2024-03-31T15:53:53.359558+0000","last_peered":"2024-03-31T15:53:53.359558+0000","last_clean":"2024-03-31T15:53:53.359558+0000","last_became_active":"2024-03-31T15:53:51.808396+0000","last_became_peered":"2024-03-31T15:53:51.808396+0000","last_unstale":"2024-03-31T15:53:53.359558+0000","last_undegraded":"2024-03-31T15:53:53.359558+0000","last_fullsized":"2024-03-31T15:53:53.359558+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920147+0000","last_change":"2024-03-31T15:53:47.770618+0000","last_active":"2024-03-31T15:53:52.920147+0000","last_peered":"2024-03-31T15:53:52.920147+0000","last_clean":"2024-03-31T15:53:52.920147+0000","last_became_active":"2024-03-31T15:53:47.770333+0000","last_became_peered":"2024-03-31T15:53:47.770333+0000","last_unstale":"2024-03-31T15:53:52.920147+0000","last_undegraded":"2024-03-31T15:53:52.920147+0000","last_fullsized":"2024-03-31T15:53:52.920147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789491+0000","last_change":"2024-03-31T15:53:46.759181+0000","last_active":"2024-03-31T15:53:52.789491+0000","last_peered":"2024-03-31T15:53:52.789491+0000","last_clean":"2024-03-31T15:53:52.789491+0000","last_became_active":"2024-03-31T15:53:46.758355+0000","last_became_peered":"2024-03-31T15:53:46.758355+0000","last_unstale":"2024-03-31T15:53:52.789491+0000","last_undegraded":"2024-03-31T15:53:52.789491+0000","last_fullsized":"2024-03-31T15:53:52.789491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306986+0000","last_change":"2024-03-31T15:53:51.796763+0000","last_active":"2024-03-31T15:53:53.306986+0000","last_peered":"2024-03-31T15:53:53.306986+0000","last_clean":"2024-03-31T15:53:53.306986+0000","last_became_active":"2024-03-31T15:53:51.796351+0000","last_became_peered":"2024-03-31T15:53:51.796351+0000","last_unstale":"2024-03-31T15:53:53.306986+0000","last_undegraded":"2024-03-31T15:53:53.306986+0000","last_fullsized":"2024-03-31T15:53:53.306986+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247797+0000","last_change":"2024-03-31T15:53:47.766443+0000","last_active":"2024-03-31T15:53:53.247797+0000","last_peered":"2024-03-31T15:53:53.247797+0000","last_clean":"2024-03-31T15:53:53.247797+0000","last_became_active":"2024-03-31T15:53:47.766107+0000","last_became_peered":"2024-03-31T15:53:47.766107+0000","last_unstale":"2024-03-31T15:53:53.247797+0000","last_undegraded":"2024-03-31T15:53:53.247797+0000","last_fullsized":"2024-03-31T15:53:53.247797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:50:47.205172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248000+0000","last_change":"2024-03-31T15:53:46.764698+0000","last_active":"2024-03-31T15:53:53.248000+0000","last_peered":"2024-03-31T15:53:53.248000+0000","last_clean":"2024-03-31T15:53:53.248000+0000","last_became_active":"2024-03-31T15:53:46.764345+0000","last_became_peered":"2024-03-31T15:53:46.764345+0000","last_unstale":"2024-03-31T15:53:53.248000+0000","last_undegraded":"2024-03-31T15:53:53.248000+0000","last_fullsized":"2024-03-31T15:53:53.248000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:42.990113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306648+0000","last_change":"2024-03-31T15:53:46.765106+0000","last_active":"2024-03-31T15:53:53.306648+0000","last_peered":"2024-03-31T15:53:53.306648+0000","last_clean":"2024-03-31T15:53:53.306648+0000","last_became_active":"2024-03-31T15:53:46.764281+0000","last_became_peered":"2024-03-31T15:53:46.764281+0000","last_unstale":"2024-03-31T15:53:53.306648+0000","last_undegraded":"2024-03-31T15:53:53.306648+0000","last_fullsized":"2024-03-31T15:53:53.306648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359310+0000","last_change":"2024-03-31T15:53:47.757501+0000","last_active":"2024-03-31T15:53:53.359310+0000","last_peered":"2024-03-31T15:53:53.359310+0000","last_clean":"2024-03-31T15:53:53.359310+0000","last_became_active":"2024-03-31T15:53:47.757160+0000","last_became_peered":"2024-03-31T15:53:47.757160+0000","last_unstale":"2024-03-31T15:53:53.359310+0000","last_undegraded":"2024-03-31T15:53:53.359310+0000","last_fullsized":"2024-03-31T15:53:53.359310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528662+0000","last_change":"2024-03-31T15:53:51.794610+0000","last_active":"2024-03-31T15:53:53.528662+0000","last_peered":"2024-03-31T15:53:53.528662+0000","last_clean":"2024-03-31T15:53:53.528662+0000","last_became_active":"2024-03-31T15:53:51.794228+0000","last_became_peered":"2024-03-31T15:53:51.794228+0000","last_unstale":"2024-03-31T15:53:53.528662+0000","last_undegraded":"2024-03-31T15:53:53.528662+0000","last_fullsized":"2024-03-31T15:53:53.528662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:55.277607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248028+0000","last_change":"2024-03-31T15:53:46.762539+0000","last_active":"2024-03-31T15:53:53.248028+0000","last_peered":"2024-03-31T15:53:53.248028+0000","last_clean":"2024-03-31T15:53:53.248028+0000","last_became_active":"2024-03-31T15:53:46.761475+0000","last_became_peered":"2024-03-31T15:53:46.761475+0000","last_unstale":"2024-03-31T15:53:53.248028+0000","last_undegraded":"2024-03-31T15:53:53.248028+0000","last_fullsized":"2024-03-31T15:53:53.248028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:36:29.320997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793114+0000","last_change":"2024-03-31T15:53:47.758390+0000","last_active":"2024-03-31T15:53:52.793114+0000","last_peered":"2024-03-31T15:53:52.793114+0000","last_clean":"2024-03-31T15:53:52.793114+0000","last_became_active":"2024-03-31T15:53:47.754114+0000","last_became_peered":"2024-03-31T15:53:47.754114+0000","last_unstale":"2024-03-31T15:53:52.793114+0000","last_undegraded":"2024-03-31T15:53:52.793114+0000","last_fullsized":"2024-03-31T15:53:52.793114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:12.842845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248390+0000","last_change":"2024-03-31T15:53:51.799618+0000","last_active":"2024-03-31T15:53:53.248390+0000","last_peered":"2024-03-31T15:53:53.248390+0000","last_clean":"2024-03-31T15:53:53.248390+0000","last_became_active":"2024-03-31T15:53:51.799171+0000","last_became_peered":"2024-03-31T15:53:51.799171+0000","last_unstale":"2024-03-31T15:53:53.248390+0000","last_undegraded":"2024-03-31T15:53:53.248390+0000","last_fullsized":"2024-03-31T15:53:53.248390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:48.155512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306339+0000","last_change":"2024-03-31T15:53:47.765092+0000","last_active":"2024-03-31T15:53:53.306339+0000","last_peered":"2024-03-31T15:53:53.306339+0000","last_clean":"2024-03-31T15:53:53.306339+0000","last_became_active":"2024-03-31T15:53:47.764701+0000","last_became_peered":"2024-03-31T15:53:47.764701+0000","last_unstale":"2024-03-31T15:53:53.306339+0000","last_undegraded":"2024-03-31T15:53:53.306339+0000","last_fullsized":"2024-03-31T15:53:53.306339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528367+0000","last_change":"2024-03-31T15:53:47.765373+0000","last_active":"2024-03-31T15:53:53.528367+0000","last_peered":"2024-03-31T15:53:53.528367+0000","last_clean":"2024-03-31T15:53:53.528367+0000","last_became_active":"2024-03-31T15:53:47.764321+0000","last_became_peered":"2024-03-31T15:53:47.764321+0000","last_unstale":"2024-03-31T15:53:53.528367+0000","last_undegraded":"2024-03-31T15:53:53.528367+0000","last_fullsized":"2024-03-31T15:53:53.528367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:19.777139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790598+0000","last_change":"2024-03-31T15:53:47.762666+0000","last_active":"2024-03-31T15:53:52.790598+0000","last_peered":"2024-03-31T15:53:52.790598+0000","last_clean":"2024-03-31T15:53:52.790598+0000","last_became_active":"2024-03-31T15:53:47.762284+0000","last_became_peered":"2024-03-31T15:53:47.762284+0000","last_unstale":"2024-03-31T15:53:52.790598+0000","last_undegraded":"2024-03-31T15:53:52.790598+0000","last_fullsized":"2024-03-31T15:53:52.790598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:44.058276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359256+0000","last_change":"2024-03-31T15:53:47.762200+0000","last_active":"2024-03-31T15:53:53.359256+0000","last_peered":"2024-03-31T15:53:53.359256+0000","last_clean":"2024-03-31T15:53:53.359256+0000","last_became_active":"2024-03-31T15:53:47.761993+0000","last_became_peered":"2024-03-31T15:53:47.761993+0000","last_unstale":"2024-03-31T15:53:53.359256+0000","last_undegraded":"2024-03-31T15:53:53.359256+0000","last_fullsized":"2024-03-31T15:53:53.359256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248436+0000","last_change":"2024-03-31T15:53:47.758308+0000","last_active":"2024-03-31T15:53:53.248436+0000","last_peered":"2024-03-31T15:53:53.248436+0000","last_clean":"2024-03-31T15:53:53.248436+0000","last_became_active":"2024-03-31T15:53:47.757996+0000","last_became_peered":"2024-03-31T15:53:47.757996+0000","last_unstale":"2024-03-31T15:53:53.248436+0000","last_undegraded":"2024-03-31T15:53:53.248436+0000","last_fullsized":"2024-03-31T15:53:53.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:24:07.611521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920834+0000","last_change":"2024-03-31T15:53:47.754477+0000","last_active":"2024-03-31T15:53:52.920834+0000","last_peered":"2024-03-31T15:53:52.920834+0000","last_clean":"2024-03-31T15:53:52.920834+0000","last_became_active":"2024-03-31T15:53:47.754071+0000","last_became_peered":"2024-03-31T15:53:47.754071+0000","last_unstale":"2024-03-31T15:53:52.920834+0000","last_undegraded":"2024-03-31T15:53:52.920834+0000","last_fullsized":"2024-03-31T15:53:52.920834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307045+0000","last_change":"2024-03-31T15:53:47.761169+0000","last_active":"2024-03-31T15:53:53.307045+0000","last_peered":"2024-03-31T15:53:53.307045+0000","last_clean":"2024-03-31T15:53:53.307045+0000","last_became_active":"2024-03-31T15:53:47.760902+0000","last_became_peered":"2024-03-31T15:53:47.760902+0000","last_unstale":"2024-03-31T15:53:53.307045+0000","last_undegraded":"2024-03-31T15:53:53.307045+0000","last_fullsized":"2024-03-31T15:53:53.307045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248647+0000","last_change":"2024-03-31T15:53:47.758495+0000","last_active":"2024-03-31T15:53:53.248647+0000","last_peered":"2024-03-31T15:53:53.248647+0000","last_clean":"2024-03-31T15:53:53.248647+0000","last_became_active":"2024-03-31T15:53:47.758146+0000","last_became_peered":"2024-03-31T15:53:47.758146+0000","last_unstale":"2024-03-31T15:53:53.248647+0000","last_undegraded":"2024-03-31T15:53:53.248647+0000","last_fullsized":"2024-03-31T15:53:53.248647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:40:45.579157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789642+0000","last_change":"2024-03-31T15:53:47.759644+0000","last_active":"2024-03-31T15:53:52.789642+0000","last_peered":"2024-03-31T15:53:52.789642+0000","last_clean":"2024-03-31T15:53:52.789642+0000","last_became_active":"2024-03-31T15:53:47.759278+0000","last_became_peered":"2024-03-31T15:53:47.759278+0000","last_unstale":"2024-03-31T15:53:52.789642+0000","last_undegraded":"2024-03-31T15:53:52.789642+0000","last_fullsized":"2024-03-31T15:53:52.789642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:36.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247837+0000","last_change":"2024-03-31T15:53:47.759026+0000","last_active":"2024-03-31T15:53:53.247837+0000","last_peered":"2024-03-31T15:53:53.247837+0000","last_clean":"2024-03-31T15:53:53.247837+0000","last_became_active":"2024-03-31T15:53:47.758752+0000","last_became_peered":"2024-03-31T15:53:47.758752+0000","last_unstale":"2024-03-31T15:53:53.247837+0000","last_undegraded":"2024-03-31T15:53:53.247837+0000","last_fullsized":"2024-03-31T15:53:53.247837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:05:25.701168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790550+0000","last_change":"2024-03-31T15:53:47.762998+0000","last_active":"2024-03-31T15:53:52.790550+0000","last_peered":"2024-03-31T15:53:52.790550+0000","last_clean":"2024-03-31T15:53:52.790550+0000","last_became_active":"2024-03-31T15:53:47.762724+0000","last_became_peered":"2024-03-31T15:53:47.762724+0000","last_unstale":"2024-03-31T15:53:52.790550+0000","last_undegraded":"2024-03-31T15:53:52.790550+0000","last_fullsized":"2024-03-31T15:53:52.790550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359161+0000","last_change":"2024-03-31T15:53:47.761675+0000","last_active":"2024-03-31T15:53:53.359161+0000","last_peered":"2024-03-31T15:53:53.359161+0000","last_clean":"2024-03-31T15:53:53.359161+0000","last_became_active":"2024-03-31T15:53:47.757931+0000","last_became_peered":"2024-03-31T15:53:47.757931+0000","last_unstale":"2024-03-31T15:53:53.359161+0000","last_undegraded":"2024-03-31T15:53:53.359161+0000","last_fullsized":"2024-03-31T15:53:53.359161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:15.515856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306768+0000","last_change":"2024-03-31T15:53:47.770705+0000","last_active":"2024-03-31T15:53:53.306768+0000","last_peered":"2024-03-31T15:53:53.306768+0000","last_clean":"2024-03-31T15:53:53.306768+0000","last_became_active":"2024-03-31T15:53:47.770485+0000","last_became_peered":"2024-03-31T15:53:47.770485+0000","last_unstale":"2024-03-31T15:53:53.306768+0000","last_undegraded":"2024-03-31T15:53:53.306768+0000","last_fullsized":"2024-03-31T15:53:53.306768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792240+0000","last_change":"2024-03-31T15:53:47.761166+0000","last_active":"2024-03-31T15:53:52.792240+0000","last_peered":"2024-03-31T15:53:52.792240+0000","last_clean":"2024-03-31T15:53:52.792240+0000","last_became_active":"2024-03-31T15:53:47.760825+0000","last_became_peered":"2024-03-31T15:53:47.760825+0000","last_unstale":"2024-03-31T15:53:52.792240+0000","last_undegraded":"2024-03-31T15:53:52.792240+0000","last_fullsized":"2024-03-31T15:53:52.792240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:56.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528829+0000","last_change":"2024-03-31T15:53:47.765006+0000","last_active":"2024-03-31T15:53:53.528829+0000","last_peered":"2024-03-31T15:53:53.528829+0000","last_clean":"2024-03-31T15:53:53.528829+0000","last_became_active":"2024-03-31T15:53:47.760192+0000","last_became_peered":"2024-03-31T15:53:47.760192+0000","last_unstale":"2024-03-31T15:53:53.528829+0000","last_undegraded":"2024-03-31T15:53:53.528829+0000","last_fullsized":"2024-03-31T15:53:53.528829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:26.202073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790075+0000","last_change":"2024-03-31T15:53:47.765871+0000","last_active":"2024-03-31T15:53:52.790075+0000","last_peered":"2024-03-31T15:53:52.790075+0000","last_clean":"2024-03-31T15:53:52.790075+0000","last_became_active":"2024-03-31T15:53:47.764257+0000","last_became_peered":"2024-03-31T15:53:47.764257+0000","last_unstale":"2024-03-31T15:53:52.790075+0000","last_undegraded":"2024-03-31T15:53:52.790075+0000","last_fullsized":"2024-03-31T15:53:52.790075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:40.058921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789357+0000","last_change":"2024-03-31T15:53:47.764320+0000","last_active":"2024-03-31T15:53:52.789357+0000","last_peered":"2024-03-31T15:53:52.789357+0000","last_clean":"2024-03-31T15:53:52.789357+0000","last_became_active":"2024-03-31T15:53:47.764067+0000","last_became_peered":"2024-03-31T15:53:47.764067+0000","last_unstale":"2024-03-31T15:53:52.789357+0000","last_undegraded":"2024-03-31T15:53:52.789357+0000","last_fullsized":"2024-03-31T15:53:52.789357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920242+0000","last_change":"2024-03-31T15:53:47.755404+0000","last_active":"2024-03-31T15:53:52.920242+0000","last_peered":"2024-03-31T15:53:52.920242+0000","last_clean":"2024-03-31T15:53:52.920242+0000","last_became_active":"2024-03-31T15:53:47.755133+0000","last_became_peered":"2024-03-31T15:53:47.755133+0000","last_unstale":"2024-03-31T15:53:52.920242+0000","last_undegraded":"2024-03-31T15:53:52.920242+0000","last_fullsized":"2024-03-31T15:53:52.920242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359051+0000","last_change":"2024-03-31T15:53:47.751494+0000","last_active":"2024-03-31T15:53:53.359051+0000","last_peered":"2024-03-31T15:53:53.359051+0000","last_clean":"2024-03-31T15:53:53.359051+0000","last_became_active":"2024-03-31T15:53:47.751219+0000","last_became_peered":"2024-03-31T15:53:47.751219+0000","last_unstale":"2024-03-31T15:53:53.359051+0000","last_undegraded":"2024-03-31T15:53:53.359051+0000","last_fullsized":"2024-03-31T15:53:53.359051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247881+0000","last_change":"2024-03-31T15:53:47.766595+0000","last_active":"2024-03-31T15:53:53.247881+0000","last_peered":"2024-03-31T15:53:53.247881+0000","last_clean":"2024-03-31T15:53:53.247881+0000","last_became_active":"2024-03-31T15:53:47.766209+0000","last_became_peered":"2024-03-31T15:53:47.766209+0000","last_unstale":"2024-03-31T15:53:53.247881+0000","last_undegraded":"2024-03-31T15:53:53.247881+0000","last_fullsized":"2024-03-31T15:53:53.247881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:51.207164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793406+0000","last_change":"2024-03-31T15:53:47.762468+0000","last_active":"2024-03-31T15:53:52.793406+0000","last_peered":"2024-03-31T15:53:52.793406+0000","last_clean":"2024-03-31T15:53:52.793406+0000","last_became_active":"2024-03-31T15:53:47.762140+0000","last_became_peered":"2024-03-31T15:53:47.762140+0000","last_unstale":"2024-03-31T15:53:52.793406+0000","last_undegraded":"2024-03-31T15:53:52.793406+0000","last_fullsized":"2024-03-31T15:53:52.793406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:04.076153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306895+0000","last_change":"2024-03-31T15:53:47.774515+0000","last_active":"2024-03-31T15:53:53.306895+0000","last_peered":"2024-03-31T15:53:53.306895+0000","last_clean":"2024-03-31T15:53:53.306895+0000","last_became_active":"2024-03-31T15:53:47.774203+0000","last_became_peered":"2024-03-31T15:53:47.774203+0000","last_unstale":"2024-03-31T15:53:53.306895+0000","last_undegraded":"2024-03-31T15:53:53.306895+0000","last_fullsized":"2024-03-31T15:53:53.306895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307016+0000","last_change":"2024-03-31T15:53:47.775151+0000","last_active":"2024-03-31T15:53:53.307016+0000","last_peered":"2024-03-31T15:53:53.307016+0000","last_clean":"2024-03-31T15:53:53.307016+0000","last_became_active":"2024-03-31T15:53:47.774661+0000","last_became_peered":"2024-03-31T15:53:47.774661+0000","last_unstale":"2024-03-31T15:53:53.307016+0000","last_undegraded":"2024-03-31T15:53:53.307016+0000","last_fullsized":"2024-03-31T15:53:53.307016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792705+0000","last_change":"2024-03-31T15:53:47.772389+0000","last_active":"2024-03-31T15:53:52.792705+0000","last_peered":"2024-03-31T15:53:52.792705+0000","last_clean":"2024-03-31T15:53:52.792705+0000","last_became_active":"2024-03-31T15:53:47.772131+0000","last_became_peered":"2024-03-31T15:53:47.772131+0000","last_unstale":"2024-03-31T15:53:52.792705+0000","last_undegraded":"2024-03-31T15:53:52.792705+0000","last_fullsized":"2024-03-31T15:53:52.792705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:35.224566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247670+0000","last_change":"2024-03-31T15:53:47.772995+0000","last_active":"2024-03-31T15:53:53.247670+0000","last_peered":"2024-03-31T15:53:53.247670+0000","last_clean":"2024-03-31T15:53:53.247670+0000","last_became_active":"2024-03-31T15:53:47.772708+0000","last_became_peered":"2024-03-31T15:53:47.772708+0000","last_unstale":"2024-03-31T15:53:53.247670+0000","last_undegraded":"2024-03-31T15:53:53.247670+0000","last_fullsized":"2024-03-31T15:53:53.247670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:12:55.991650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791467+0000","last_change":"2024-03-31T15:53:47.766440+0000","last_active":"2024-03-31T15:53:52.791467+0000","last_peered":"2024-03-31T15:53:52.791467+0000","last_clean":"2024-03-31T15:53:52.791467+0000","last_became_active":"2024-03-31T15:53:47.766187+0000","last_became_peered":"2024-03-31T15:53:47.766187+0000","last_unstale":"2024-03-31T15:53:52.791467+0000","last_undegraded":"2024-03-31T15:53:52.791467+0000","last_fullsized":"2024-03-31T15:53:52.791467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:53.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790499+0000","last_change":"2024-03-31T15:53:47.778379+0000","last_active":"2024-03-31T15:53:52.790499+0000","last_peered":"2024-03-31T15:53:52.790499+0000","last_clean":"2024-03-31T15:53:52.790499+0000","last_became_active":"2024-03-31T15:53:47.778049+0000","last_became_peered":"2024-03-31T15:53:47.778049+0000","last_unstale":"2024-03-31T15:53:52.790499+0000","last_undegraded":"2024-03-31T15:53:52.790499+0000","last_fullsized":"2024-03-31T15:53:52.790499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791659+0000","last_change":"2024-03-31T15:53:47.761184+0000","last_active":"2024-03-31T15:53:52.791659+0000","last_peered":"2024-03-31T15:53:52.791659+0000","last_clean":"2024-03-31T15:53:52.791659+0000","last_became_active":"2024-03-31T15:53:47.760863+0000","last_became_peered":"2024-03-31T15:53:47.760863+0000","last_unstale":"2024-03-31T15:53:52.791659+0000","last_undegraded":"2024-03-31T15:53:52.791659+0000","last_fullsized":"2024-03-31T15:53:52.791659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:08:38.336677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789652+0000","last_change":"2024-03-31T15:53:47.766801+0000","last_active":"2024-03-31T15:53:52.789652+0000","last_peered":"2024-03-31T15:53:52.789652+0000","last_clean":"2024-03-31T15:53:52.789652+0000","last_became_active":"2024-03-31T15:53:47.766512+0000","last_became_peered":"2024-03-31T15:53:47.766512+0000","last_unstale":"2024-03-31T15:53:52.789652+0000","last_undegraded":"2024-03-31T15:53:52.789652+0000","last_fullsized":"2024-03-31T15:53:52.789652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:25.798405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359475+0000","last_change":"2024-03-31T15:53:47.756732+0000","last_active":"2024-03-31T15:53:53.359475+0000","last_peered":"2024-03-31T15:53:53.359475+0000","last_clean":"2024-03-31T15:53:53.359475+0000","last_became_active":"2024-03-31T15:53:47.756466+0000","last_became_peered":"2024-03-31T15:53:47.756466+0000","last_unstale":"2024-03-31T15:53:53.359475+0000","last_undegraded":"2024-03-31T15:53:53.359475+0000","last_fullsized":"2024-03-31T15:53:53.359475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921162+0000","last_change":"2024-03-31T15:53:47.763716+0000","last_active":"2024-03-31T15:53:52.921162+0000","last_peered":"2024-03-31T15:53:52.921162+0000","last_clean":"2024-03-31T15:53:52.921162+0000","last_became_active":"2024-03-31T15:53:47.763310+0000","last_became_peered":"2024-03-31T15:53:47.763310+0000","last_unstale":"2024-03-31T15:53:52.921162+0000","last_undegraded":"2024-03-31T15:53:52.921162+0000","last_fullsized":"2024-03-31T15:53:52.921162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792492+0000","last_change":"2024-03-31T15:53:47.765680+0000","last_active":"2024-03-31T15:53:52.792492+0000","last_peered":"2024-03-31T15:53:52.792492+0000","last_clean":"2024-03-31T15:53:52.792492+0000","last_became_active":"2024-03-31T15:53:47.765516+0000","last_became_peered":"2024-03-31T15:53:47.765516+0000","last_unstale":"2024-03-31T15:53:52.792492+0000","last_undegraded":"2024-03-31T15:53:52.792492+0000","last_fullsized":"2024-03-31T15:53:52.792492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:44:09.409693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509451,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":102303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":102143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":856064,"data_stored":693093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:54:17.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:54:17.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.998+0000 7f6af1600640 1 Processor -- start 2024-03-31T15:54:17.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:17.998+0000 7f6af1600640 1 -- start start 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6af1600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6aec06d7c0 0x7f6aec15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6af1600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aec15f780 con 0x7f6aec06c7b0 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6af1600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aec15f1c0 con 0x7f6aec15e1b0 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6af1600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6aec1622b0 con 0x7f6aec06d7c0 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6aec06d7c0 0x7f6aec15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6ae3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6aec06c7b0 0x7f6aec06d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34028/0 (socket says 172.21.15.90:34028) 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6aec06d7c0 0x7f6aec15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51622/0 (socket says 172.21.15.90:51622) 2024-03-31T15:54:17.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6ae3e00640 1 -- 172.21.15.90:0/848035697 learned_addr learned my addr 172.21.15.90:0/848035697 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:17.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 -- 172.21.15.90:0/848035697 >> v1:172.21.15.102:6789/0 conn(0x7f6aec15e1b0 legacy=0x7f6aec162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 -- 172.21.15.90:0/848035697 >> v1:172.21.15.90:6789/0 conn(0x7f6aec06c7b0 legacy=0x7f6aec06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:17.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 -- 172.21.15.90:0/848035697 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec15f4a0 con 0x7f6aec06d7c0 2024-03-31T15:54:17.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.002+0000 7f6aeb400640 1 --2- 172.21.15.90:0/848035697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06d7c0 0x7f6aec15e590 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009d90 tx=0x7f6ae401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92277b4a7383247f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:17.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aea000640 1 -- 172.21.15.90:0/848035697 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40b2070 con 0x7f6aec06d7c0 2024-03-31T15:54:17.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aea000640 1 -- 172.21.15.90:0/848035697 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae40a5970 con 0x7f6aec06d7c0 2024-03-31T15:54:17.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aea000640 1 -- 172.21.15.90:0/848035697 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40a5d40 con 0x7f6aec06d7c0 2024-03-31T15:54:18.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- 172.21.15.90:0/848035697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06d7c0 msgr2=0x7f6aec15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 --2- 172.21.15.90:0/848035697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06d7c0 0x7f6aec15e590 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4009d90 tx=0x7f6ae401c8b0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- 172.21.15.90:0/848035697 shutdown_connections 2024-03-31T15:54:18.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 --2- 172.21.15.90:0/848035697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06d7c0 0x7f6aec15e590 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- 172.21.15.90:0/848035697 >> 172.21.15.90:0/848035697 conn(0x7f6aec09aa40 msgr2=0x7f6aec09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:18.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- 172.21.15.90:0/848035697 shutdown_connections 2024-03-31T15:54:18.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- 172.21.15.90:0/848035697 wait complete. 2024-03-31T15:54:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 Processor -- start 2024-03-31T15:54:18.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- start start 2024-03-31T15:54:18.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aec06d7c0 0x7f6aec0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6aec15e1b0 0x7f6aec156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec0ce7a0 con 0x7f6aec06c7b0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec0d2a50 con 0x7f6aec06d7c0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6af1600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6aec0ce920 con 0x7f6aec15e1b0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aeaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51628/0 (socket says 172.21.15.90:51628) 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.006+0000 7f6aeaa00640 1 -- 172.21.15.90:0/663506423 learned_addr learned my addr 172.21.15.90:0/663506423 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeb400640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6aec15e1b0 0x7f6aec156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6ae3e00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aec06d7c0 0x7f6aec0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 -- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6aec15e1b0 msgr2=0x7f6aec156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6aec15e1b0 0x7f6aec156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 -- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aec06d7c0 msgr2=0x7f6aec0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aec06d7c0 0x7f6aec0ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 -- 172.21.15.90:0/663506423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec0ca640 con 0x7f6aec06c7b0 2024-03-31T15:54:18.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6aeaa00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00c370 tx=0x7f6adc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0052d0 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc005470 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc007260 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec15d480 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec15c6c0 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aec0686d0 con 0x7f6aec06c7b0 2024-03-31T15:54:18.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.010+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6adc007400 con 0x7f6aec06c7b0 2024-03-31T15:54:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.014+0000 7f6ae2a00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 0x7f6ac806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.014+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6adc086250 con 0x7f6aec06c7b0 2024-03-31T15:54:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.014+0000 7f6ae3e00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 0x7f6ac806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.014+0000 7f6ae3e00640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 0x7f6ac806a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6aec06e120 tx=0x7f6ad4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.018+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6aec0ca640 con 0x7f6aec06c7b0 2024-03-31T15:54:18.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.182+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6aec0b7de0 con 0x7f6ac8068380 2024-03-31T15:54:18.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.190+0000 7f6ae2a00640 1 -- 172.21.15.90:0/663506423 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332537 (secure 0 0 0) 0x7f6aec0b7de0 con 0x7f6ac8068380 2024-03-31T15:54:18.191 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:18.193 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 msgr2=0x7f6ac806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 0x7f6ac806a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6aec06e120 tx=0x7f6ad4009040 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 msgr2=0x7f6aec15bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00c370 tx=0x7f6adc00c830 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 shutdown_connections 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ac8068380 0x7f6ac806a840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6aec15e1b0 0x7f6aec156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aec06d7c0 0x7f6aec0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 --2- 172.21.15.90:0/663506423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6aec06c7b0 0x7f6aec15bcb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 >> 172.21.15.90:0/663506423 conn(0x7f6aec09aa40 msgr2=0x7f6aec0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:18.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 shutdown_connections 2024-03-31T15:54:18.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.194+0000 7f6af1600640 1 -- 172.21.15.90:0/663506423 wait complete. 2024-03-31T15:54:18.209 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-31T15:54:17.222591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219416,"kb_used_data":3240,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755304,"statfs":{"total":773094113280,"available":772869431296,"internally_reserved":0,"allocated":3317760,"data_stored":2007160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003781"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359793+0000","last_change":"2024-03-31T15:53:46.757375+0000","last_active":"2024-03-31T15:53:53.359793+0000","last_peered":"2024-03-31T15:53:53.359793+0000","last_clean":"2024-03-31T15:53:53.359793+0000","last_became_active":"2024-03-31T15:53:46.757092+0000","last_became_peered":"2024-03-31T15:53:46.757092+0000","last_unstale":"2024-03-31T15:53:53.359793+0000","last_undegraded":"2024-03-31T15:53:53.359793+0000","last_fullsized":"2024-03-31T15:53:53.359793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248347+0000","last_change":"2024-03-31T15:53:47.766865+0000","last_active":"2024-03-31T15:53:53.248347+0000","last_peered":"2024-03-31T15:53:53.248347+0000","last_clean":"2024-03-31T15:53:53.248347+0000","last_became_active":"2024-03-31T15:53:47.766202+0000","last_became_peered":"2024-03-31T15:53:47.766202+0000","last_unstale":"2024-03-31T15:53:53.248347+0000","last_undegraded":"2024-03-31T15:53:53.248347+0000","last_fullsized":"2024-03-31T15:53:53.248347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:03:31.997737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247733+0000","last_change":"2024-03-31T15:53:51.804152+0000","last_active":"2024-03-31T15:53:53.247733+0000","last_peered":"2024-03-31T15:53:53.247733+0000","last_clean":"2024-03-31T15:53:53.247733+0000","last_became_active":"2024-03-31T15:53:51.802511+0000","last_became_peered":"2024-03-31T15:53:51.802511+0000","last_unstale":"2024-03-31T15:53:53.247733+0000","last_undegraded":"2024-03-31T15:53:53.247733+0000","last_fullsized":"2024-03-31T15:53:53.247733+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:38.965494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248547+0000","last_change":"2024-03-31T15:53:46.763714+0000","last_active":"2024-03-31T15:53:53.248547+0000","last_peered":"2024-03-31T15:53:53.248547+0000","last_clean":"2024-03-31T15:53:53.248547+0000","last_became_active":"2024-03-31T15:53:46.763376+0000","last_became_peered":"2024-03-31T15:53:46.763376+0000","last_unstale":"2024-03-31T15:53:53.248547+0000","last_undegraded":"2024-03-31T15:53:53.248547+0000","last_fullsized":"2024-03-31T15:53:53.248547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359675+0000","last_change":"2024-03-31T15:53:47.771438+0000","last_active":"2024-03-31T15:53:53.359675+0000","last_peered":"2024-03-31T15:53:53.359675+0000","last_clean":"2024-03-31T15:53:53.359675+0000","last_became_active":"2024-03-31T15:53:47.771175+0000","last_became_peered":"2024-03-31T15:53:47.771175+0000","last_unstale":"2024-03-31T15:53:53.359675+0000","last_undegraded":"2024-03-31T15:53:53.359675+0000","last_fullsized":"2024-03-31T15:53:53.359675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792499+0000","last_change":"2024-03-31T15:53:51.797656+0000","last_active":"2024-03-31T15:53:52.792499+0000","last_peered":"2024-03-31T15:53:52.792499+0000","last_clean":"2024-03-31T15:53:52.792499+0000","last_became_active":"2024-03-31T15:53:51.797356+0000","last_became_peered":"2024-03-31T15:53:51.797356+0000","last_unstale":"2024-03-31T15:53:52.792499+0000","last_undegraded":"2024-03-31T15:53:52.792499+0000","last_fullsized":"2024-03-31T15:53:52.792499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:05.027492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792090+0000","last_change":"2024-03-31T15:53:46.764787+0000","last_active":"2024-03-31T15:53:52.792090+0000","last_peered":"2024-03-31T15:53:52.792090+0000","last_clean":"2024-03-31T15:53:52.792090+0000","last_became_active":"2024-03-31T15:53:46.764484+0000","last_became_peered":"2024-03-31T15:53:46.764484+0000","last_unstale":"2024-03-31T15:53:52.792090+0000","last_undegraded":"2024-03-31T15:53:52.792090+0000","last_fullsized":"2024-03-31T15:53:52.792090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:22:15.262251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789729+0000","last_change":"2024-03-31T15:53:47.773460+0000","last_active":"2024-03-31T15:53:52.789729+0000","last_peered":"2024-03-31T15:53:52.789729+0000","last_clean":"2024-03-31T15:53:52.789729+0000","last_became_active":"2024-03-31T15:53:47.773148+0000","last_became_peered":"2024-03-31T15:53:47.773148+0000","last_unstale":"2024-03-31T15:53:52.789729+0000","last_undegraded":"2024-03-31T15:53:52.789729+0000","last_fullsized":"2024-03-31T15:53:52.789729+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:54.396848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793233+0000","last_change":"2024-03-31T15:53:51.799743+0000","last_active":"2024-03-31T15:53:52.793233+0000","last_peered":"2024-03-31T15:53:52.793233+0000","last_clean":"2024-03-31T15:53:52.793233+0000","last_became_active":"2024-03-31T15:53:51.799332+0000","last_became_peered":"2024-03-31T15:53:51.799332+0000","last_unstale":"2024-03-31T15:53:52.793233+0000","last_undegraded":"2024-03-31T15:53:52.793233+0000","last_fullsized":"2024-03-31T15:53:52.793233+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:10:58.431101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791817+0000","last_change":"2024-03-31T15:53:46.765429+0000","last_active":"2024-03-31T15:53:52.791817+0000","last_peered":"2024-03-31T15:53:52.791817+0000","last_clean":"2024-03-31T15:53:52.791817+0000","last_became_active":"2024-03-31T15:53:46.765129+0000","last_became_peered":"2024-03-31T15:53:46.765129+0000","last_unstale":"2024-03-31T15:53:52.791817+0000","last_undegraded":"2024-03-31T15:53:52.791817+0000","last_fullsized":"2024-03-31T15:53:52.791817+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:00:22.460897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790207+0000","last_change":"2024-03-31T15:53:47.770281+0000","last_active":"2024-03-31T15:53:52.790207+0000","last_peered":"2024-03-31T15:53:52.790207+0000","last_clean":"2024-03-31T15:53:52.790207+0000","last_became_active":"2024-03-31T15:53:47.769991+0000","last_became_peered":"2024-03-31T15:53:47.769991+0000","last_unstale":"2024-03-31T15:53:52.790207+0000","last_undegraded":"2024-03-31T15:53:52.790207+0000","last_fullsized":"2024-03-31T15:53:52.790207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:20.227687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792734+0000","last_change":"2024-03-31T15:53:51.797162+0000","last_active":"2024-03-31T15:53:52.792734+0000","last_peered":"2024-03-31T15:53:52.792734+0000","last_clean":"2024-03-31T15:53:52.792734+0000","last_became_active":"2024-03-31T15:53:51.796702+0000","last_became_peered":"2024-03-31T15:53:51.796702+0000","last_unstale":"2024-03-31T15:53:52.792734+0000","last_undegraded":"2024-03-31T15:53:52.792734+0000","last_fullsized":"2024-03-31T15:53:52.792734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:40.593467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306348+0000","last_change":"2024-03-31T15:53:47.762689+0000","last_active":"2024-03-31T15:53:53.306348+0000","last_peered":"2024-03-31T15:53:53.306348+0000","last_clean":"2024-03-31T15:53:53.306348+0000","last_became_active":"2024-03-31T15:53:47.762427+0000","last_became_peered":"2024-03-31T15:53:47.762427+0000","last_unstale":"2024-03-31T15:53:53.306348+0000","last_undegraded":"2024-03-31T15:53:53.306348+0000","last_fullsized":"2024-03-31T15:53:53.306348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791816+0000","last_change":"2024-03-31T15:53:46.757859+0000","last_active":"2024-03-31T15:53:52.791816+0000","last_peered":"2024-03-31T15:53:52.791816+0000","last_clean":"2024-03-31T15:53:52.791816+0000","last_became_active":"2024-03-31T15:53:46.757618+0000","last_became_peered":"2024-03-31T15:53:46.757618+0000","last_unstale":"2024-03-31T15:53:52.791816+0000","last_undegraded":"2024-03-31T15:53:52.791816+0000","last_fullsized":"2024-03-31T15:53:52.791816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:08:18.634688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793519+0000","last_change":"2024-03-31T15:53:51.797915+0000","last_active":"2024-03-31T15:53:52.793519+0000","last_peered":"2024-03-31T15:53:52.793519+0000","last_clean":"2024-03-31T15:53:52.793519+0000","last_became_active":"2024-03-31T15:53:51.797515+0000","last_became_peered":"2024-03-31T15:53:51.797515+0000","last_unstale":"2024-03-31T15:53:52.793519+0000","last_undegraded":"2024-03-31T15:53:52.793519+0000","last_fullsized":"2024-03-31T15:53:52.793519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247998+0000","last_change":"2024-03-31T15:53:47.765185+0000","last_active":"2024-03-31T15:53:53.247998+0000","last_peered":"2024-03-31T15:53:53.247998+0000","last_clean":"2024-03-31T15:53:53.247998+0000","last_became_active":"2024-03-31T15:53:47.764777+0000","last_became_peered":"2024-03-31T15:53:47.764777+0000","last_unstale":"2024-03-31T15:53:53.247998+0000","last_undegraded":"2024-03-31T15:53:53.247998+0000","last_fullsized":"2024-03-31T15:53:53.247998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:08:23.224004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920654+0000","last_change":"2024-03-31T15:53:46.762534+0000","last_active":"2024-03-31T15:53:52.920654+0000","last_peered":"2024-03-31T15:53:52.920654+0000","last_clean":"2024-03-31T15:53:52.920654+0000","last_became_active":"2024-03-31T15:53:46.762081+0000","last_became_peered":"2024-03-31T15:53:46.762081+0000","last_unstale":"2024-03-31T15:53:52.920654+0000","last_undegraded":"2024-03-31T15:53:52.920654+0000","last_fullsized":"2024-03-31T15:53:52.920654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793296+0000","last_change":"2024-03-31T15:53:51.802744+0000","last_active":"2024-03-31T15:53:52.793296+0000","last_peered":"2024-03-31T15:53:52.793296+0000","last_clean":"2024-03-31T15:53:52.793296+0000","last_became_active":"2024-03-31T15:53:51.802328+0000","last_became_peered":"2024-03-31T15:53:51.802328+0000","last_unstale":"2024-03-31T15:53:52.793296+0000","last_undegraded":"2024-03-31T15:53:52.793296+0000","last_fullsized":"2024-03-31T15:53:52.793296+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:34:00.099017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247904+0000","last_change":"2024-03-31T15:53:47.751398+0000","last_active":"2024-03-31T15:53:53.247904+0000","last_peered":"2024-03-31T15:53:53.247904+0000","last_clean":"2024-03-31T15:53:53.247904+0000","last_became_active":"2024-03-31T15:53:47.750758+0000","last_became_peered":"2024-03-31T15:53:47.750758+0000","last_unstale":"2024-03-31T15:53:53.247904+0000","last_undegraded":"2024-03-31T15:53:53.247904+0000","last_fullsized":"2024-03-31T15:53:53.247904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:44:31.275051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920257+0000","last_change":"2024-03-31T15:53:46.755068+0000","last_active":"2024-03-31T15:53:52.920257+0000","last_peered":"2024-03-31T15:53:52.920257+0000","last_clean":"2024-03-31T15:53:52.920257+0000","last_became_active":"2024-03-31T15:53:46.754754+0000","last_became_peered":"2024-03-31T15:53:46.754754+0000","last_unstale":"2024-03-31T15:53:52.920257+0000","last_undegraded":"2024-03-31T15:53:52.920257+0000","last_fullsized":"2024-03-31T15:53:52.920257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921340+0000","last_change":"2024-03-31T15:53:51.803182+0000","last_active":"2024-03-31T15:53:52.921340+0000","last_peered":"2024-03-31T15:53:52.921340+0000","last_clean":"2024-03-31T15:53:52.921340+0000","last_became_active":"2024-03-31T15:53:51.802710+0000","last_became_peered":"2024-03-31T15:53:51.802710+0000","last_unstale":"2024-03-31T15:53:52.921340+0000","last_undegraded":"2024-03-31T15:53:52.921340+0000","last_fullsized":"2024-03-31T15:53:52.921340+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:40.165393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792965+0000","last_change":"2024-03-31T15:53:47.755358+0000","last_active":"2024-03-31T15:53:52.792965+0000","last_peered":"2024-03-31T15:53:52.792965+0000","last_clean":"2024-03-31T15:53:52.792965+0000","last_became_active":"2024-03-31T15:53:47.754487+0000","last_became_peered":"2024-03-31T15:53:47.754487+0000","last_unstale":"2024-03-31T15:53:52.792965+0000","last_undegraded":"2024-03-31T15:53:52.792965+0000","last_fullsized":"2024-03-31T15:53:52.792965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:53.949585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791156+0000","last_change":"2024-03-31T15:53:46.755464+0000","last_active":"2024-03-31T15:53:52.791156+0000","last_peered":"2024-03-31T15:53:52.791156+0000","last_clean":"2024-03-31T15:53:52.791156+0000","last_became_active":"2024-03-31T15:53:46.754626+0000","last_became_peered":"2024-03-31T15:53:46.754626+0000","last_unstale":"2024-03-31T15:53:52.791156+0000","last_undegraded":"2024-03-31T15:53:52.791156+0000","last_fullsized":"2024-03-31T15:53:52.791156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:18.769264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793394+0000","last_change":"2024-03-31T15:53:51.803474+0000","last_active":"2024-03-31T15:53:52.793394+0000","last_peered":"2024-03-31T15:53:52.793394+0000","last_clean":"2024-03-31T15:53:52.793394+0000","last_became_active":"2024-03-31T15:53:51.803014+0000","last_became_peered":"2024-03-31T15:53:51.803014+0000","last_unstale":"2024-03-31T15:53:52.793394+0000","last_undegraded":"2024-03-31T15:53:52.793394+0000","last_fullsized":"2024-03-31T15:53:52.793394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:11.321777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359741+0000","last_change":"2024-03-31T15:53:47.754929+0000","last_active":"2024-03-31T15:53:53.359741+0000","last_peered":"2024-03-31T15:53:53.359741+0000","last_clean":"2024-03-31T15:53:53.359741+0000","last_became_active":"2024-03-31T15:53:47.754625+0000","last_became_peered":"2024-03-31T15:53:47.754625+0000","last_unstale":"2024-03-31T15:53:53.359741+0000","last_undegraded":"2024-03-31T15:53:53.359741+0000","last_fullsized":"2024-03-31T15:53:53.359741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307339+0000","last_change":"2024-03-31T15:53:46.757784+0000","last_active":"2024-03-31T15:53:53.307339+0000","last_peered":"2024-03-31T15:53:53.307339+0000","last_clean":"2024-03-31T15:53:53.307339+0000","last_became_active":"2024-03-31T15:53:46.757342+0000","last_became_peered":"2024-03-31T15:53:46.757342+0000","last_unstale":"2024-03-31T15:53:53.307339+0000","last_undegraded":"2024-03-31T15:53:53.307339+0000","last_fullsized":"2024-03-31T15:53:53.307339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792725+0000","last_change":"2024-03-31T15:53:51.800310+0000","last_active":"2024-03-31T15:53:52.792725+0000","last_peered":"2024-03-31T15:53:52.792725+0000","last_clean":"2024-03-31T15:53:52.792725+0000","last_became_active":"2024-03-31T15:53:51.798223+0000","last_became_peered":"2024-03-31T15:53:51.798223+0000","last_unstale":"2024-03-31T15:53:52.792725+0000","last_undegraded":"2024-03-31T15:53:52.792725+0000","last_fullsized":"2024-03-31T15:53:52.792725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:48.092530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248692+0000","last_change":"2024-03-31T15:53:47.754983+0000","last_active":"2024-03-31T15:53:53.248692+0000","last_peered":"2024-03-31T15:53:53.248692+0000","last_clean":"2024-03-31T15:53:53.248692+0000","last_became_active":"2024-03-31T15:53:47.754734+0000","last_became_peered":"2024-03-31T15:53:47.754734+0000","last_unstale":"2024-03-31T15:53:53.248692+0000","last_undegraded":"2024-03-31T15:53:53.248692+0000","last_fullsized":"2024-03-31T15:53:53.248692+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:31:24.931497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921179+0000","last_change":"2024-03-31T15:53:46.763360+0000","last_active":"2024-03-31T15:53:52.921179+0000","last_peered":"2024-03-31T15:53:52.921179+0000","last_clean":"2024-03-31T15:53:52.921179+0000","last_became_active":"2024-03-31T15:53:46.763005+0000","last_became_peered":"2024-03-31T15:53:46.763005+0000","last_unstale":"2024-03-31T15:53:52.921179+0000","last_undegraded":"2024-03-31T15:53:52.921179+0000","last_fullsized":"2024-03-31T15:53:52.921179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:30.435282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247663+0000","last_change":"2024-03-31T15:53:51.800117+0000","last_active":"2024-03-31T15:53:53.247663+0000","last_peered":"2024-03-31T15:53:53.247663+0000","last_clean":"2024-03-31T15:53:53.247663+0000","last_became_active":"2024-03-31T15:53:51.799217+0000","last_became_peered":"2024-03-31T15:53:51.799217+0000","last_unstale":"2024-03-31T15:53:53.247663+0000","last_undegraded":"2024-03-31T15:53:53.247663+0000","last_fullsized":"2024-03-31T15:53:53.247663+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:39.258820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920855+0000","last_change":"2024-03-31T15:53:47.754428+0000","last_active":"2024-03-31T15:53:52.920855+0000","last_peered":"2024-03-31T15:53:52.920855+0000","last_clean":"2024-03-31T15:53:52.920855+0000","last_became_active":"2024-03-31T15:53:47.753085+0000","last_became_peered":"2024-03-31T15:53:47.753085+0000","last_unstale":"2024-03-31T15:53:52.920855+0000","last_undegraded":"2024-03-31T15:53:52.920855+0000","last_fullsized":"2024-03-31T15:53:52.920855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792251+0000","last_change":"2024-03-31T15:53:46.763802+0000","last_active":"2024-03-31T15:53:52.792251+0000","last_peered":"2024-03-31T15:53:52.792251+0000","last_clean":"2024-03-31T15:53:52.792251+0000","last_became_active":"2024-03-31T15:53:46.762025+0000","last_became_peered":"2024-03-31T15:53:46.762025+0000","last_unstale":"2024-03-31T15:53:52.792251+0000","last_undegraded":"2024-03-31T15:53:52.792251+0000","last_fullsized":"2024-03-31T15:53:52.792251+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:50.235479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359173+0000","last_change":"2024-03-31T15:53:51.796387+0000","last_active":"2024-03-31T15:53:53.359173+0000","last_peered":"2024-03-31T15:53:53.359173+0000","last_clean":"2024-03-31T15:53:53.359173+0000","last_became_active":"2024-03-31T15:53:51.795890+0000","last_became_peered":"2024-03-31T15:53:51.795890+0000","last_unstale":"2024-03-31T15:53:53.359173+0000","last_undegraded":"2024-03-31T15:53:53.359173+0000","last_fullsized":"2024-03-31T15:53:53.359173+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790297+0000","last_change":"2024-03-31T15:53:47.765672+0000","last_active":"2024-03-31T15:53:52.790297+0000","last_peered":"2024-03-31T15:53:52.790297+0000","last_clean":"2024-03-31T15:53:52.790297+0000","last_became_active":"2024-03-31T15:53:47.765481+0000","last_became_peered":"2024-03-31T15:53:47.765481+0000","last_unstale":"2024-03-31T15:53:52.790297+0000","last_undegraded":"2024-03-31T15:53:52.790297+0000","last_fullsized":"2024-03-31T15:53:52.790297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:14.332486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920943+0000","last_change":"2024-03-31T15:53:46.752123+0000","last_active":"2024-03-31T15:53:52.920943+0000","last_peered":"2024-03-31T15:53:52.920943+0000","last_clean":"2024-03-31T15:53:52.920943+0000","last_became_active":"2024-03-31T15:53:46.751752+0000","last_became_peered":"2024-03-31T15:53:46.751752+0000","last_unstale":"2024-03-31T15:53:52.920943+0000","last_undegraded":"2024-03-31T15:53:52.920943+0000","last_fullsized":"2024-03-31T15:53:52.920943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920358+0000","last_change":"2024-03-31T15:53:51.798289+0000","last_active":"2024-03-31T15:53:52.920358+0000","last_peered":"2024-03-31T15:53:52.920358+0000","last_clean":"2024-03-31T15:53:52.920358+0000","last_became_active":"2024-03-31T15:53:51.797843+0000","last_became_peered":"2024-03-31T15:53:51.797843+0000","last_unstale":"2024-03-31T15:53:52.920358+0000","last_undegraded":"2024-03-31T15:53:52.920358+0000","last_fullsized":"2024-03-31T15:53:52.920358+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793832+0000","last_change":"2024-03-31T15:53:47.746865+0000","last_active":"2024-03-31T15:53:52.793832+0000","last_peered":"2024-03-31T15:53:52.793832+0000","last_clean":"2024-03-31T15:53:52.793832+0000","last_became_active":"2024-03-31T15:53:47.746098+0000","last_became_peered":"2024-03-31T15:53:47.746098+0000","last_unstale":"2024-03-31T15:53:52.793832+0000","last_undegraded":"2024-03-31T15:53:52.793832+0000","last_fullsized":"2024-03-31T15:53:52.793832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:17:31.443157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790702+0000","last_change":"2024-03-31T15:53:46.753263+0000","last_active":"2024-03-31T15:53:52.790702+0000","last_peered":"2024-03-31T15:53:52.790702+0000","last_clean":"2024-03-31T15:53:52.790702+0000","last_became_active":"2024-03-31T15:53:46.751947+0000","last_became_peered":"2024-03-31T15:53:46.751947+0000","last_unstale":"2024-03-31T15:53:52.790702+0000","last_undegraded":"2024-03-31T15:53:52.790702+0000","last_fullsized":"2024-03-31T15:53:52.790702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920882+0000","last_change":"2024-03-31T15:53:51.797078+0000","last_active":"2024-03-31T15:53:52.920882+0000","last_peered":"2024-03-31T15:53:52.920882+0000","last_clean":"2024-03-31T15:53:52.920882+0000","last_became_active":"2024-03-31T15:53:51.796451+0000","last_became_peered":"2024-03-31T15:53:51.796451+0000","last_unstale":"2024-03-31T15:53:52.920882+0000","last_undegraded":"2024-03-31T15:53:52.920882+0000","last_fullsized":"2024-03-31T15:53:52.920882+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920377+0000","last_change":"2024-03-31T15:53:47.752652+0000","last_active":"2024-03-31T15:53:52.920377+0000","last_peered":"2024-03-31T15:53:52.920377+0000","last_clean":"2024-03-31T15:53:52.920377+0000","last_became_active":"2024-03-31T15:53:47.752105+0000","last_became_peered":"2024-03-31T15:53:47.752105+0000","last_unstale":"2024-03-31T15:53:52.920377+0000","last_undegraded":"2024-03-31T15:53:52.920377+0000","last_fullsized":"2024-03-31T15:53:52.920377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528393+0000","last_change":"2024-03-31T15:53:46.753745+0000","last_active":"2024-03-31T15:53:53.528393+0000","last_peered":"2024-03-31T15:53:53.528393+0000","last_clean":"2024-03-31T15:53:53.528393+0000","last_became_active":"2024-03-31T15:53:46.753423+0000","last_became_peered":"2024-03-31T15:53:46.753423+0000","last_unstale":"2024-03-31T15:53:53.528393+0000","last_undegraded":"2024-03-31T15:53:53.528393+0000","last_fullsized":"2024-03-31T15:53:53.528393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:52:24.485978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921329+0000","last_change":"2024-03-31T15:53:51.802368+0000","last_active":"2024-03-31T15:53:52.921329+0000","last_peered":"2024-03-31T15:53:52.921329+0000","last_clean":"2024-03-31T15:53:52.921329+0000","last_became_active":"2024-03-31T15:53:51.801946+0000","last_became_peered":"2024-03-31T15:53:51.801946+0000","last_unstale":"2024-03-31T15:53:52.921329+0000","last_undegraded":"2024-03-31T15:53:52.921329+0000","last_fullsized":"2024-03-31T15:53:52.921329+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.358775+0000","last_change":"2024-03-31T15:53:47.745539+0000","last_active":"2024-03-31T15:53:53.358775+0000","last_peered":"2024-03-31T15:53:53.358775+0000","last_clean":"2024-03-31T15:53:53.358775+0000","last_became_active":"2024-03-31T15:53:47.745265+0000","last_became_peered":"2024-03-31T15:53:47.745265+0000","last_unstale":"2024-03-31T15:53:53.358775+0000","last_undegraded":"2024-03-31T15:53:53.358775+0000","last_fullsized":"2024-03-31T15:53:53.358775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:12:22.550188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791245+0000","last_change":"2024-03-31T15:53:46.761890+0000","last_active":"2024-03-31T15:53:52.791245+0000","last_peered":"2024-03-31T15:53:52.791245+0000","last_clean":"2024-03-31T15:53:52.791245+0000","last_became_active":"2024-03-31T15:53:46.761417+0000","last_became_peered":"2024-03-31T15:53:46.761417+0000","last_unstale":"2024-03-31T15:53:52.791245+0000","last_undegraded":"2024-03-31T15:53:52.791245+0000","last_fullsized":"2024-03-31T15:53:52.791245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:25.194198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359685+0000","last_change":"2024-03-31T15:53:51.796610+0000","last_active":"2024-03-31T15:53:53.359685+0000","last_peered":"2024-03-31T15:53:53.359685+0000","last_clean":"2024-03-31T15:53:53.359685+0000","last_became_active":"2024-03-31T15:53:51.796112+0000","last_became_peered":"2024-03-31T15:53:51.796112+0000","last_unstale":"2024-03-31T15:53:53.359685+0000","last_undegraded":"2024-03-31T15:53:53.359685+0000","last_fullsized":"2024-03-31T15:53:53.359685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:14:50.793500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247912+0000","last_change":"2024-03-31T15:53:47.768681+0000","last_active":"2024-03-31T15:53:53.247912+0000","last_peered":"2024-03-31T15:53:53.247912+0000","last_clean":"2024-03-31T15:53:53.247912+0000","last_became_active":"2024-03-31T15:53:47.768416+0000","last_became_peered":"2024-03-31T15:53:47.768416+0000","last_unstale":"2024-03-31T15:53:53.247912+0000","last_undegraded":"2024-03-31T15:53:53.247912+0000","last_fullsized":"2024-03-31T15:53:53.247912+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:23:02.383327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789855+0000","last_change":"2024-03-31T15:53:46.760553+0000","last_active":"2024-03-31T15:53:52.789855+0000","last_peered":"2024-03-31T15:53:52.789855+0000","last_clean":"2024-03-31T15:53:52.789855+0000","last_became_active":"2024-03-31T15:53:46.760258+0000","last_became_peered":"2024-03-31T15:53:46.760258+0000","last_unstale":"2024-03-31T15:53:52.789855+0000","last_undegraded":"2024-03-31T15:53:52.789855+0000","last_fullsized":"2024-03-31T15:53:52.789855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791305+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:53:52.791305+0000","last_peered":"2024-03-31T15:53:52.791305+0000","last_clean":"2024-03-31T15:53:52.791305+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:53:52.791305+0000","last_undegraded":"2024-03-31T15:53:52.791305+0000","last_fullsized":"2024-03-31T15:53:52.791305+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306820+0000","last_change":"2024-03-31T15:53:51.789901+0000","last_active":"2024-03-31T15:53:53.306820+0000","last_peered":"2024-03-31T15:53:53.306820+0000","last_clean":"2024-03-31T15:53:53.306820+0000","last_became_active":"2024-03-31T15:53:51.789529+0000","last_became_peered":"2024-03-31T15:53:51.789529+0000","last_unstale":"2024-03-31T15:53:53.306820+0000","last_undegraded":"2024-03-31T15:53:53.306820+0000","last_fullsized":"2024-03-31T15:53:53.306820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791956+0000","last_change":"2024-03-31T15:53:47.764710+0000","last_active":"2024-03-31T15:53:52.791956+0000","last_peered":"2024-03-31T15:53:52.791956+0000","last_clean":"2024-03-31T15:53:52.791956+0000","last_became_active":"2024-03-31T15:53:47.764408+0000","last_became_peered":"2024-03-31T15:53:47.764408+0000","last_unstale":"2024-03-31T15:53:52.791956+0000","last_undegraded":"2024-03-31T15:53:52.791956+0000","last_fullsized":"2024-03-31T15:53:52.791956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:14:45.512051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528719+0000","last_change":"2024-03-31T15:53:46.757343+0000","last_active":"2024-03-31T15:53:53.528719+0000","last_peered":"2024-03-31T15:53:53.528719+0000","last_clean":"2024-03-31T15:53:53.528719+0000","last_became_active":"2024-03-31T15:53:46.757028+0000","last_became_peered":"2024-03-31T15:53:46.757028+0000","last_unstale":"2024-03-31T15:53:53.528719+0000","last_undegraded":"2024-03-31T15:53:53.528719+0000","last_fullsized":"2024-03-31T15:53:53.528719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:25:19.521516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306461+0000","last_change":"2024-03-31T15:53:51.799696+0000","last_active":"2024-03-31T15:53:53.306461+0000","last_peered":"2024-03-31T15:53:53.306461+0000","last_clean":"2024-03-31T15:53:53.306461+0000","last_became_active":"2024-03-31T15:53:51.799041+0000","last_became_peered":"2024-03-31T15:53:51.799041+0000","last_unstale":"2024-03-31T15:53:53.306461+0000","last_undegraded":"2024-03-31T15:53:53.306461+0000","last_fullsized":"2024-03-31T15:53:53.306461+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:30:24.261940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789786+0000","last_change":"2024-03-31T15:53:47.767189+0000","last_active":"2024-03-31T15:53:52.789786+0000","last_peered":"2024-03-31T15:53:52.789786+0000","last_clean":"2024-03-31T15:53:52.789786+0000","last_became_active":"2024-03-31T15:53:47.766941+0000","last_became_peered":"2024-03-31T15:53:47.766941+0000","last_unstale":"2024-03-31T15:53:52.789786+0000","last_undegraded":"2024-03-31T15:53:52.789786+0000","last_fullsized":"2024-03-31T15:53:52.789786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:56.064763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921010+0000","last_change":"2024-03-31T15:53:46.755784+0000","last_active":"2024-03-31T15:53:52.921010+0000","last_peered":"2024-03-31T15:53:52.921010+0000","last_clean":"2024-03-31T15:53:52.921010+0000","last_became_active":"2024-03-31T15:53:46.755518+0000","last_became_peered":"2024-03-31T15:53:46.755518+0000","last_unstale":"2024-03-31T15:53:52.921010+0000","last_undegraded":"2024-03-31T15:53:52.921010+0000","last_fullsized":"2024-03-31T15:53:52.921010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:03.068278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359266+0000","last_change":"2024-03-31T15:53:51.805030+0000","last_active":"2024-03-31T15:53:53.359266+0000","last_peered":"2024-03-31T15:53:53.359266+0000","last_clean":"2024-03-31T15:53:53.359266+0000","last_became_active":"2024-03-31T15:53:51.804608+0000","last_became_peered":"2024-03-31T15:53:51.804608+0000","last_unstale":"2024-03-31T15:53:53.359266+0000","last_undegraded":"2024-03-31T15:53:53.359266+0000","last_fullsized":"2024-03-31T15:53:53.359266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792768+0000","last_change":"2024-03-31T15:53:47.763067+0000","last_active":"2024-03-31T15:53:52.792768+0000","last_peered":"2024-03-31T15:53:52.792768+0000","last_clean":"2024-03-31T15:53:52.792768+0000","last_became_active":"2024-03-31T15:53:47.762801+0000","last_became_peered":"2024-03-31T15:53:47.762801+0000","last_unstale":"2024-03-31T15:53:52.792768+0000","last_undegraded":"2024-03-31T15:53:52.792768+0000","last_fullsized":"2024-03-31T15:53:52.792768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:34.931240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528883+0000","last_change":"2024-03-31T15:53:46.763307+0000","last_active":"2024-03-31T15:53:53.528883+0000","last_peered":"2024-03-31T15:53:53.528883+0000","last_clean":"2024-03-31T15:53:53.528883+0000","last_became_active":"2024-03-31T15:53:46.763083+0000","last_became_peered":"2024-03-31T15:53:46.763083+0000","last_unstale":"2024-03-31T15:53:53.528883+0000","last_undegraded":"2024-03-31T15:53:53.528883+0000","last_fullsized":"2024-03-31T15:53:53.528883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:35:35.431822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789351+0000","last_change":"2024-03-31T15:53:51.792009+0000","last_active":"2024-03-31T15:53:52.789351+0000","last_peered":"2024-03-31T15:53:52.789351+0000","last_clean":"2024-03-31T15:53:52.789351+0000","last_became_active":"2024-03-31T15:53:51.791660+0000","last_became_peered":"2024-03-31T15:53:51.791660+0000","last_unstale":"2024-03-31T15:53:52.789351+0000","last_undegraded":"2024-03-31T15:53:52.789351+0000","last_fullsized":"2024-03-31T15:53:52.789351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:11:17.783876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792256+0000","last_change":"2024-03-31T15:53:47.771025+0000","last_active":"2024-03-31T15:53:52.792256+0000","last_peered":"2024-03-31T15:53:52.792256+0000","last_clean":"2024-03-31T15:53:52.792256+0000","last_became_active":"2024-03-31T15:53:47.769254+0000","last_became_peered":"2024-03-31T15:53:47.769254+0000","last_unstale":"2024-03-31T15:53:52.792256+0000","last_undegraded":"2024-03-31T15:53:52.792256+0000","last_fullsized":"2024-03-31T15:53:52.792256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:01.503832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793608+0000","last_change":"2024-03-31T15:53:46.759316+0000","last_active":"2024-03-31T15:53:52.793608+0000","last_peered":"2024-03-31T15:53:52.793608+0000","last_clean":"2024-03-31T15:53:52.793608+0000","last_became_active":"2024-03-31T15:53:46.758973+0000","last_became_peered":"2024-03-31T15:53:46.758973+0000","last_unstale":"2024-03-31T15:53:52.793608+0000","last_undegraded":"2024-03-31T15:53:52.793608+0000","last_fullsized":"2024-03-31T15:53:52.793608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:39.824106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792641+0000","last_change":"2024-03-31T15:53:51.795201+0000","last_active":"2024-03-31T15:53:52.792641+0000","last_peered":"2024-03-31T15:53:52.792641+0000","last_clean":"2024-03-31T15:53:52.792641+0000","last_became_active":"2024-03-31T15:53:51.794775+0000","last_became_peered":"2024-03-31T15:53:51.794775+0000","last_unstale":"2024-03-31T15:53:52.792641+0000","last_undegraded":"2024-03-31T15:53:52.792641+0000","last_fullsized":"2024-03-31T15:53:52.792641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:11.574086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791547+0000","last_change":"2024-03-31T15:53:47.760294+0000","last_active":"2024-03-31T15:53:52.791547+0000","last_peered":"2024-03-31T15:53:52.791547+0000","last_clean":"2024-03-31T15:53:52.791547+0000","last_became_active":"2024-03-31T15:53:47.760011+0000","last_became_peered":"2024-03-31T15:53:47.760011+0000","last_unstale":"2024-03-31T15:53:52.791547+0000","last_undegraded":"2024-03-31T15:53:52.791547+0000","last_fullsized":"2024-03-31T15:53:52.791547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:11.819602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528298+0000","last_change":"2024-03-31T15:53:46.759221+0000","last_active":"2024-03-31T15:53:53.528298+0000","last_peered":"2024-03-31T15:53:53.528298+0000","last_clean":"2024-03-31T15:53:53.528298+0000","last_became_active":"2024-03-31T15:53:46.758926+0000","last_became_peered":"2024-03-31T15:53:46.758926+0000","last_unstale":"2024-03-31T15:53:53.528298+0000","last_undegraded":"2024-03-31T15:53:53.528298+0000","last_fullsized":"2024-03-31T15:53:53.528298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:11:40.380065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359571+0000","last_change":"2024-03-31T15:53:51.794069+0000","last_active":"2024-03-31T15:53:53.359571+0000","last_peered":"2024-03-31T15:53:53.359571+0000","last_clean":"2024-03-31T15:53:53.359571+0000","last_became_active":"2024-03-31T15:53:51.793665+0000","last_became_peered":"2024-03-31T15:53:51.793665+0000","last_unstale":"2024-03-31T15:53:53.359571+0000","last_undegraded":"2024-03-31T15:53:53.359571+0000","last_fullsized":"2024-03-31T15:53:53.359571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:56.254946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306482+0000","last_change":"2024-03-31T15:53:47.763992+0000","last_active":"2024-03-31T15:53:53.306482+0000","last_peered":"2024-03-31T15:53:53.306482+0000","last_clean":"2024-03-31T15:53:53.306482+0000","last_became_active":"2024-03-31T15:53:47.763743+0000","last_became_peered":"2024-03-31T15:53:47.763743+0000","last_unstale":"2024-03-31T15:53:53.306482+0000","last_undegraded":"2024-03-31T15:53:53.306482+0000","last_fullsized":"2024-03-31T15:53:53.306482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306483+0000","last_change":"2024-03-31T15:53:46.764342+0000","last_active":"2024-03-31T15:53:53.306483+0000","last_peered":"2024-03-31T15:53:53.306483+0000","last_clean":"2024-03-31T15:53:53.306483+0000","last_became_active":"2024-03-31T15:53:46.763064+0000","last_became_peered":"2024-03-31T15:53:46.763064+0000","last_unstale":"2024-03-31T15:53:53.306483+0000","last_undegraded":"2024-03-31T15:53:53.306483+0000","last_fullsized":"2024-03-31T15:53:53.306483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248662+0000","last_change":"2024-03-31T15:53:51.802982+0000","last_active":"2024-03-31T15:53:53.248662+0000","last_peered":"2024-03-31T15:53:53.248662+0000","last_clean":"2024-03-31T15:53:53.248662+0000","last_became_active":"2024-03-31T15:53:51.802619+0000","last_became_peered":"2024-03-31T15:53:51.802619+0000","last_unstale":"2024-03-31T15:53:53.248662+0000","last_undegraded":"2024-03-31T15:53:53.248662+0000","last_fullsized":"2024-03-31T15:53:53.248662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:12:18.187743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791793+0000","last_change":"2024-03-31T15:53:47.763046+0000","last_active":"2024-03-31T15:53:52.791793+0000","last_peered":"2024-03-31T15:53:52.791793+0000","last_clean":"2024-03-31T15:53:52.791793+0000","last_became_active":"2024-03-31T15:53:47.762791+0000","last_became_peered":"2024-03-31T15:53:47.762791+0000","last_unstale":"2024-03-31T15:53:52.791793+0000","last_undegraded":"2024-03-31T15:53:52.791793+0000","last_fullsized":"2024-03-31T15:53:52.791793+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:32:01.910382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790646+0000","last_change":"2024-03-31T15:53:46.765124+0000","last_active":"2024-03-31T15:53:52.790646+0000","last_peered":"2024-03-31T15:53:52.790646+0000","last_clean":"2024-03-31T15:53:52.790646+0000","last_became_active":"2024-03-31T15:53:46.764856+0000","last_became_peered":"2024-03-31T15:53:46.764856+0000","last_unstale":"2024-03-31T15:53:52.790646+0000","last_undegraded":"2024-03-31T15:53:52.790646+0000","last_fullsized":"2024-03-31T15:53:52.790646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:07.248237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248292+0000","last_change":"2024-03-31T15:53:51.807271+0000","last_active":"2024-03-31T15:53:53.248292+0000","last_peered":"2024-03-31T15:53:53.248292+0000","last_clean":"2024-03-31T15:53:53.248292+0000","last_became_active":"2024-03-31T15:53:51.806909+0000","last_became_peered":"2024-03-31T15:53:51.806909+0000","last_unstale":"2024-03-31T15:53:53.248292+0000","last_undegraded":"2024-03-31T15:53:53.248292+0000","last_fullsized":"2024-03-31T15:53:53.248292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:14:12.015472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359081+0000","last_change":"2024-03-31T15:53:47.762410+0000","last_active":"2024-03-31T15:53:53.359081+0000","last_peered":"2024-03-31T15:53:53.359081+0000","last_clean":"2024-03-31T15:53:53.359081+0000","last_became_active":"2024-03-31T15:53:47.762145+0000","last_became_peered":"2024-03-31T15:53:47.762145+0000","last_unstale":"2024-03-31T15:53:53.359081+0000","last_undegraded":"2024-03-31T15:53:53.359081+0000","last_fullsized":"2024-03-31T15:53:53.359081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359350+0000","last_change":"2024-03-31T15:53:46.756308+0000","last_active":"2024-03-31T15:53:53.359350+0000","last_peered":"2024-03-31T15:53:53.359350+0000","last_clean":"2024-03-31T15:53:53.359350+0000","last_became_active":"2024-03-31T15:53:46.756010+0000","last_became_peered":"2024-03-31T15:53:46.756010+0000","last_unstale":"2024-03-31T15:53:53.359350+0000","last_undegraded":"2024-03-31T15:53:53.359350+0000","last_fullsized":"2024-03-31T15:53:53.359350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:30:06.680148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790084+0000","last_change":"2024-03-31T15:53:51.806516+0000","last_active":"2024-03-31T15:53:52.790084+0000","last_peered":"2024-03-31T15:53:52.790084+0000","last_clean":"2024-03-31T15:53:52.790084+0000","last_became_active":"2024-03-31T15:53:51.805985+0000","last_became_peered":"2024-03-31T15:53:51.805985+0000","last_unstale":"2024-03-31T15:53:52.790084+0000","last_undegraded":"2024-03-31T15:53:52.790084+0000","last_fullsized":"2024-03-31T15:53:52.790084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:12:27.917062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791963+0000","last_change":"2024-03-31T15:53:47.768000+0000","last_active":"2024-03-31T15:53:52.791963+0000","last_peered":"2024-03-31T15:53:52.791963+0000","last_clean":"2024-03-31T15:53:52.791963+0000","last_became_active":"2024-03-31T15:53:47.767710+0000","last_became_peered":"2024-03-31T15:53:47.767710+0000","last_unstale":"2024-03-31T15:53:52.791963+0000","last_undegraded":"2024-03-31T15:53:52.791963+0000","last_fullsized":"2024-03-31T15:53:52.791963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:12:35.607484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:54:02.857882+0000","last_change":"2024-03-31T15:53:46.764364+0000","last_active":"2024-03-31T15:54:02.857882+0000","last_peered":"2024-03-31T15:54:02.857882+0000","last_clean":"2024-03-31T15:54:02.857882+0000","last_became_active":"2024-03-31T15:53:46.763234+0000","last_became_peered":"2024-03-31T15:53:46.763234+0000","last_unstale":"2024-03-31T15:54:02.857882+0000","last_undegraded":"2024-03-31T15:54:02.857882+0000","last_fullsized":"2024-03-31T15:54:02.857882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920563+0000","last_change":"2024-03-31T15:53:51.794393+0000","last_active":"2024-03-31T15:53:52.920563+0000","last_peered":"2024-03-31T15:53:52.920563+0000","last_clean":"2024-03-31T15:53:52.920563+0000","last_became_active":"2024-03-31T15:53:51.793984+0000","last_became_peered":"2024-03-31T15:53:51.793984+0000","last_unstale":"2024-03-31T15:53:52.920563+0000","last_undegraded":"2024-03-31T15:53:52.920563+0000","last_fullsized":"2024-03-31T15:53:52.920563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793726+0000","last_change":"2024-03-31T15:53:47.759150+0000","last_active":"2024-03-31T15:53:52.793726+0000","last_peered":"2024-03-31T15:53:52.793726+0000","last_clean":"2024-03-31T15:53:52.793726+0000","last_became_active":"2024-03-31T15:53:47.757846+0000","last_became_peered":"2024-03-31T15:53:47.757846+0000","last_unstale":"2024-03-31T15:53:52.793726+0000","last_undegraded":"2024-03-31T15:53:52.793726+0000","last_fullsized":"2024-03-31T15:53:52.793726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:16.643130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920932+0000","last_change":"2024-03-31T15:53:46.763645+0000","last_active":"2024-03-31T15:53:52.920932+0000","last_peered":"2024-03-31T15:53:52.920932+0000","last_clean":"2024-03-31T15:53:52.920932+0000","last_became_active":"2024-03-31T15:53:46.763370+0000","last_became_peered":"2024-03-31T15:53:46.763370+0000","last_unstale":"2024-03-31T15:53:52.920932+0000","last_undegraded":"2024-03-31T15:53:52.920932+0000","last_fullsized":"2024-03-31T15:53:52.920932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:51:15.635255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793545+0000","last_change":"2024-03-31T15:53:51.804406+0000","last_active":"2024-03-31T15:53:52.793545+0000","last_peered":"2024-03-31T15:53:52.793545+0000","last_clean":"2024-03-31T15:53:52.793545+0000","last_became_active":"2024-03-31T15:53:51.803887+0000","last_became_peered":"2024-03-31T15:53:51.803887+0000","last_unstale":"2024-03-31T15:53:52.793545+0000","last_undegraded":"2024-03-31T15:53:52.793545+0000","last_fullsized":"2024-03-31T15:53:52.793545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:44.093175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359778+0000","last_change":"2024-03-31T15:53:47.771467+0000","last_active":"2024-03-31T15:53:53.359778+0000","last_peered":"2024-03-31T15:53:53.359778+0000","last_clean":"2024-03-31T15:53:53.359778+0000","last_became_active":"2024-03-31T15:53:47.771267+0000","last_became_peered":"2024-03-31T15:53:47.771267+0000","last_unstale":"2024-03-31T15:53:53.359778+0000","last_undegraded":"2024-03-31T15:53:53.359778+0000","last_fullsized":"2024-03-31T15:53:53.359778+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790236+0000","last_change":"2024-03-31T15:53:46.758836+0000","last_active":"2024-03-31T15:53:52.790236+0000","last_peered":"2024-03-31T15:53:52.790236+0000","last_clean":"2024-03-31T15:53:52.790236+0000","last_became_active":"2024-03-31T15:53:46.758359+0000","last_became_peered":"2024-03-31T15:53:46.758359+0000","last_unstale":"2024-03-31T15:53:52.790236+0000","last_undegraded":"2024-03-31T15:53:52.790236+0000","last_fullsized":"2024-03-31T15:53:52.790236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:42:52.941710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247843+0000","last_change":"2024-03-31T15:53:51.800437+0000","last_active":"2024-03-31T15:53:53.247843+0000","last_peered":"2024-03-31T15:53:53.247843+0000","last_clean":"2024-03-31T15:53:53.247843+0000","last_became_active":"2024-03-31T15:53:51.799728+0000","last_became_peered":"2024-03-31T15:53:51.799728+0000","last_unstale":"2024-03-31T15:53:53.247843+0000","last_undegraded":"2024-03-31T15:53:53.247843+0000","last_fullsized":"2024-03-31T15:53:53.247843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:57:45.272362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790191+0000","last_change":"2024-03-31T15:53:47.765037+0000","last_active":"2024-03-31T15:53:52.790191+0000","last_peered":"2024-03-31T15:53:52.790191+0000","last_clean":"2024-03-31T15:53:52.790191+0000","last_became_active":"2024-03-31T15:53:47.764764+0000","last_became_peered":"2024-03-31T15:53:47.764764+0000","last_unstale":"2024-03-31T15:53:52.790191+0000","last_undegraded":"2024-03-31T15:53:52.790191+0000","last_fullsized":"2024-03-31T15:53:52.790191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:05:07.287524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307175+0000","last_change":"2024-03-31T15:53:46.765922+0000","last_active":"2024-03-31T15:53:53.307175+0000","last_peered":"2024-03-31T15:53:53.307175+0000","last_clean":"2024-03-31T15:53:53.307175+0000","last_became_active":"2024-03-31T15:53:46.765530+0000","last_became_peered":"2024-03-31T15:53:46.765530+0000","last_unstale":"2024-03-31T15:53:53.307175+0000","last_undegraded":"2024-03-31T15:53:53.307175+0000","last_fullsized":"2024-03-31T15:53:53.307175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:11.834970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247784+0000","last_change":"2024-03-31T15:53:51.803591+0000","last_active":"2024-03-31T15:53:53.247784+0000","last_peered":"2024-03-31T15:53:53.247784+0000","last_clean":"2024-03-31T15:53:53.247784+0000","last_became_active":"2024-03-31T15:53:51.803139+0000","last_became_peered":"2024-03-31T15:53:51.803139+0000","last_unstale":"2024-03-31T15:53:53.247784+0000","last_undegraded":"2024-03-31T15:53:53.247784+0000","last_fullsized":"2024-03-31T15:53:53.247784+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:41:08.680601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359558+0000","last_change":"2024-03-31T15:53:51.809399+0000","last_active":"2024-03-31T15:53:53.359558+0000","last_peered":"2024-03-31T15:53:53.359558+0000","last_clean":"2024-03-31T15:53:53.359558+0000","last_became_active":"2024-03-31T15:53:51.808396+0000","last_became_peered":"2024-03-31T15:53:51.808396+0000","last_unstale":"2024-03-31T15:53:53.359558+0000","last_undegraded":"2024-03-31T15:53:53.359558+0000","last_fullsized":"2024-03-31T15:53:53.359558+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:00:18.366739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920147+0000","last_change":"2024-03-31T15:53:47.770618+0000","last_active":"2024-03-31T15:53:52.920147+0000","last_peered":"2024-03-31T15:53:52.920147+0000","last_clean":"2024-03-31T15:53:52.920147+0000","last_became_active":"2024-03-31T15:53:47.770333+0000","last_became_peered":"2024-03-31T15:53:47.770333+0000","last_unstale":"2024-03-31T15:53:52.920147+0000","last_undegraded":"2024-03-31T15:53:52.920147+0000","last_fullsized":"2024-03-31T15:53:52.920147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789491+0000","last_change":"2024-03-31T15:53:46.759181+0000","last_active":"2024-03-31T15:53:52.789491+0000","last_peered":"2024-03-31T15:53:52.789491+0000","last_clean":"2024-03-31T15:53:52.789491+0000","last_became_active":"2024-03-31T15:53:46.758355+0000","last_became_peered":"2024-03-31T15:53:46.758355+0000","last_unstale":"2024-03-31T15:53:52.789491+0000","last_undegraded":"2024-03-31T15:53:52.789491+0000","last_fullsized":"2024-03-31T15:53:52.789491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:57:07.517981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306986+0000","last_change":"2024-03-31T15:53:51.796763+0000","last_active":"2024-03-31T15:53:53.306986+0000","last_peered":"2024-03-31T15:53:53.306986+0000","last_clean":"2024-03-31T15:53:53.306986+0000","last_became_active":"2024-03-31T15:53:51.796351+0000","last_became_peered":"2024-03-31T15:53:51.796351+0000","last_unstale":"2024-03-31T15:53:53.306986+0000","last_undegraded":"2024-03-31T15:53:53.306986+0000","last_fullsized":"2024-03-31T15:53:53.306986+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:29.832659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247797+0000","last_change":"2024-03-31T15:53:47.766443+0000","last_active":"2024-03-31T15:53:53.247797+0000","last_peered":"2024-03-31T15:53:53.247797+0000","last_clean":"2024-03-31T15:53:53.247797+0000","last_became_active":"2024-03-31T15:53:47.766107+0000","last_became_peered":"2024-03-31T15:53:47.766107+0000","last_unstale":"2024-03-31T15:53:53.247797+0000","last_undegraded":"2024-03-31T15:53:53.247797+0000","last_fullsized":"2024-03-31T15:53:53.247797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:50:47.205172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248000+0000","last_change":"2024-03-31T15:53:46.764698+0000","last_active":"2024-03-31T15:53:53.248000+0000","last_peered":"2024-03-31T15:53:53.248000+0000","last_clean":"2024-03-31T15:53:53.248000+0000","last_became_active":"2024-03-31T15:53:46.764345+0000","last_became_peered":"2024-03-31T15:53:46.764345+0000","last_unstale":"2024-03-31T15:53:53.248000+0000","last_undegraded":"2024-03-31T15:53:53.248000+0000","last_fullsized":"2024-03-31T15:53:53.248000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:57:42.990113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306648+0000","last_change":"2024-03-31T15:53:46.765106+0000","last_active":"2024-03-31T15:53:53.306648+0000","last_peered":"2024-03-31T15:53:53.306648+0000","last_clean":"2024-03-31T15:53:53.306648+0000","last_became_active":"2024-03-31T15:53:46.764281+0000","last_became_peered":"2024-03-31T15:53:46.764281+0000","last_unstale":"2024-03-31T15:53:53.306648+0000","last_undegraded":"2024-03-31T15:53:53.306648+0000","last_fullsized":"2024-03-31T15:53:53.306648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:22.842889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359310+0000","last_change":"2024-03-31T15:53:47.757501+0000","last_active":"2024-03-31T15:53:53.359310+0000","last_peered":"2024-03-31T15:53:53.359310+0000","last_clean":"2024-03-31T15:53:53.359310+0000","last_became_active":"2024-03-31T15:53:47.757160+0000","last_became_peered":"2024-03-31T15:53:47.757160+0000","last_unstale":"2024-03-31T15:53:53.359310+0000","last_undegraded":"2024-03-31T15:53:53.359310+0000","last_fullsized":"2024-03-31T15:53:53.359310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528662+0000","last_change":"2024-03-31T15:53:51.794610+0000","last_active":"2024-03-31T15:53:53.528662+0000","last_peered":"2024-03-31T15:53:53.528662+0000","last_clean":"2024-03-31T15:53:53.528662+0000","last_became_active":"2024-03-31T15:53:51.794228+0000","last_became_peered":"2024-03-31T15:53:51.794228+0000","last_unstale":"2024-03-31T15:53:53.528662+0000","last_undegraded":"2024-03-31T15:53:53.528662+0000","last_fullsized":"2024-03-31T15:53:53.528662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:30:55.277607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248028+0000","last_change":"2024-03-31T15:53:46.762539+0000","last_active":"2024-03-31T15:53:53.248028+0000","last_peered":"2024-03-31T15:53:53.248028+0000","last_clean":"2024-03-31T15:53:53.248028+0000","last_became_active":"2024-03-31T15:53:46.761475+0000","last_became_peered":"2024-03-31T15:53:46.761475+0000","last_unstale":"2024-03-31T15:53:53.248028+0000","last_undegraded":"2024-03-31T15:53:53.248028+0000","last_fullsized":"2024-03-31T15:53:53.248028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:45.719307+0000","last_clean_scrub_stamp":"2024-03-31T15:53:45.719307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:36:29.320997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793114+0000","last_change":"2024-03-31T15:53:47.758390+0000","last_active":"2024-03-31T15:53:52.793114+0000","last_peered":"2024-03-31T15:53:52.793114+0000","last_clean":"2024-03-31T15:53:52.793114+0000","last_became_active":"2024-03-31T15:53:47.754114+0000","last_became_peered":"2024-03-31T15:53:47.754114+0000","last_unstale":"2024-03-31T15:53:52.793114+0000","last_undegraded":"2024-03-31T15:53:52.793114+0000","last_fullsized":"2024-03-31T15:53:52.793114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:27:12.842845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248390+0000","last_change":"2024-03-31T15:53:51.799618+0000","last_active":"2024-03-31T15:53:53.248390+0000","last_peered":"2024-03-31T15:53:53.248390+0000","last_clean":"2024-03-31T15:53:53.248390+0000","last_became_active":"2024-03-31T15:53:51.799171+0000","last_became_peered":"2024-03-31T15:53:51.799171+0000","last_unstale":"2024-03-31T15:53:53.248390+0000","last_undegraded":"2024-03-31T15:53:53.248390+0000","last_fullsized":"2024-03-31T15:53:53.248390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:50.761435+0000","last_clean_scrub_stamp":"2024-03-31T15:53:50.761435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:51:48.155512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306339+0000","last_change":"2024-03-31T15:53:47.765092+0000","last_active":"2024-03-31T15:53:53.306339+0000","last_peered":"2024-03-31T15:53:53.306339+0000","last_clean":"2024-03-31T15:53:53.306339+0000","last_became_active":"2024-03-31T15:53:47.764701+0000","last_became_peered":"2024-03-31T15:53:47.764701+0000","last_unstale":"2024-03-31T15:53:53.306339+0000","last_undegraded":"2024-03-31T15:53:53.306339+0000","last_fullsized":"2024-03-31T15:53:53.306339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528367+0000","last_change":"2024-03-31T15:53:47.765373+0000","last_active":"2024-03-31T15:53:53.528367+0000","last_peered":"2024-03-31T15:53:53.528367+0000","last_clean":"2024-03-31T15:53:53.528367+0000","last_became_active":"2024-03-31T15:53:47.764321+0000","last_became_peered":"2024-03-31T15:53:47.764321+0000","last_unstale":"2024-03-31T15:53:53.528367+0000","last_undegraded":"2024-03-31T15:53:53.528367+0000","last_fullsized":"2024-03-31T15:53:53.528367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:06:19.777139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790598+0000","last_change":"2024-03-31T15:53:47.762666+0000","last_active":"2024-03-31T15:53:52.790598+0000","last_peered":"2024-03-31T15:53:52.790598+0000","last_clean":"2024-03-31T15:53:52.790598+0000","last_became_active":"2024-03-31T15:53:47.762284+0000","last_became_peered":"2024-03-31T15:53:47.762284+0000","last_unstale":"2024-03-31T15:53:52.790598+0000","last_undegraded":"2024-03-31T15:53:52.790598+0000","last_fullsized":"2024-03-31T15:53:52.790598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:46:44.058276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359256+0000","last_change":"2024-03-31T15:53:47.762200+0000","last_active":"2024-03-31T15:53:53.359256+0000","last_peered":"2024-03-31T15:53:53.359256+0000","last_clean":"2024-03-31T15:53:53.359256+0000","last_became_active":"2024-03-31T15:53:47.761993+0000","last_became_peered":"2024-03-31T15:53:47.761993+0000","last_unstale":"2024-03-31T15:53:53.359256+0000","last_undegraded":"2024-03-31T15:53:53.359256+0000","last_fullsized":"2024-03-31T15:53:53.359256+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:24:39.152719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248436+0000","last_change":"2024-03-31T15:53:47.758308+0000","last_active":"2024-03-31T15:53:53.248436+0000","last_peered":"2024-03-31T15:53:53.248436+0000","last_clean":"2024-03-31T15:53:53.248436+0000","last_became_active":"2024-03-31T15:53:47.757996+0000","last_became_peered":"2024-03-31T15:53:47.757996+0000","last_unstale":"2024-03-31T15:53:53.248436+0000","last_undegraded":"2024-03-31T15:53:53.248436+0000","last_fullsized":"2024-03-31T15:53:53.248436+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:24:07.611521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920834+0000","last_change":"2024-03-31T15:53:47.754477+0000","last_active":"2024-03-31T15:53:52.920834+0000","last_peered":"2024-03-31T15:53:52.920834+0000","last_clean":"2024-03-31T15:53:52.920834+0000","last_became_active":"2024-03-31T15:53:47.754071+0000","last_became_peered":"2024-03-31T15:53:47.754071+0000","last_unstale":"2024-03-31T15:53:52.920834+0000","last_undegraded":"2024-03-31T15:53:52.920834+0000","last_fullsized":"2024-03-31T15:53:52.920834+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307045+0000","last_change":"2024-03-31T15:53:47.761169+0000","last_active":"2024-03-31T15:53:53.307045+0000","last_peered":"2024-03-31T15:53:53.307045+0000","last_clean":"2024-03-31T15:53:53.307045+0000","last_became_active":"2024-03-31T15:53:47.760902+0000","last_became_peered":"2024-03-31T15:53:47.760902+0000","last_unstale":"2024-03-31T15:53:53.307045+0000","last_undegraded":"2024-03-31T15:53:53.307045+0000","last_fullsized":"2024-03-31T15:53:53.307045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.248647+0000","last_change":"2024-03-31T15:53:47.758495+0000","last_active":"2024-03-31T15:53:53.248647+0000","last_peered":"2024-03-31T15:53:53.248647+0000","last_clean":"2024-03-31T15:53:53.248647+0000","last_became_active":"2024-03-31T15:53:47.758146+0000","last_became_peered":"2024-03-31T15:53:47.758146+0000","last_unstale":"2024-03-31T15:53:53.248647+0000","last_undegraded":"2024-03-31T15:53:53.248647+0000","last_fullsized":"2024-03-31T15:53:53.248647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:40:45.579157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789642+0000","last_change":"2024-03-31T15:53:47.759644+0000","last_active":"2024-03-31T15:53:52.789642+0000","last_peered":"2024-03-31T15:53:52.789642+0000","last_clean":"2024-03-31T15:53:52.789642+0000","last_became_active":"2024-03-31T15:53:47.759278+0000","last_became_peered":"2024-03-31T15:53:47.759278+0000","last_unstale":"2024-03-31T15:53:52.789642+0000","last_undegraded":"2024-03-31T15:53:52.789642+0000","last_fullsized":"2024-03-31T15:53:52.789642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:31:36.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247837+0000","last_change":"2024-03-31T15:53:47.759026+0000","last_active":"2024-03-31T15:53:53.247837+0000","last_peered":"2024-03-31T15:53:53.247837+0000","last_clean":"2024-03-31T15:53:53.247837+0000","last_became_active":"2024-03-31T15:53:47.758752+0000","last_became_peered":"2024-03-31T15:53:47.758752+0000","last_unstale":"2024-03-31T15:53:53.247837+0000","last_undegraded":"2024-03-31T15:53:53.247837+0000","last_fullsized":"2024-03-31T15:53:53.247837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:05:25.701168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790550+0000","last_change":"2024-03-31T15:53:47.762998+0000","last_active":"2024-03-31T15:53:52.790550+0000","last_peered":"2024-03-31T15:53:52.790550+0000","last_clean":"2024-03-31T15:53:52.790550+0000","last_became_active":"2024-03-31T15:53:47.762724+0000","last_became_peered":"2024-03-31T15:53:47.762724+0000","last_unstale":"2024-03-31T15:53:52.790550+0000","last_undegraded":"2024-03-31T15:53:52.790550+0000","last_fullsized":"2024-03-31T15:53:52.790550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359161+0000","last_change":"2024-03-31T15:53:47.761675+0000","last_active":"2024-03-31T15:53:53.359161+0000","last_peered":"2024-03-31T15:53:53.359161+0000","last_clean":"2024-03-31T15:53:53.359161+0000","last_became_active":"2024-03-31T15:53:47.757931+0000","last_became_peered":"2024-03-31T15:53:47.757931+0000","last_unstale":"2024-03-31T15:53:53.359161+0000","last_undegraded":"2024-03-31T15:53:53.359161+0000","last_fullsized":"2024-03-31T15:53:53.359161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:34:15.515856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306768+0000","last_change":"2024-03-31T15:53:47.770705+0000","last_active":"2024-03-31T15:53:53.306768+0000","last_peered":"2024-03-31T15:53:53.306768+0000","last_clean":"2024-03-31T15:53:53.306768+0000","last_became_active":"2024-03-31T15:53:47.770485+0000","last_became_peered":"2024-03-31T15:53:47.770485+0000","last_unstale":"2024-03-31T15:53:53.306768+0000","last_undegraded":"2024-03-31T15:53:53.306768+0000","last_fullsized":"2024-03-31T15:53:53.306768+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:37.912156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792240+0000","last_change":"2024-03-31T15:53:47.761166+0000","last_active":"2024-03-31T15:53:52.792240+0000","last_peered":"2024-03-31T15:53:52.792240+0000","last_clean":"2024-03-31T15:53:52.792240+0000","last_became_active":"2024-03-31T15:53:47.760825+0000","last_became_peered":"2024-03-31T15:53:47.760825+0000","last_unstale":"2024-03-31T15:53:52.792240+0000","last_undegraded":"2024-03-31T15:53:52.792240+0000","last_fullsized":"2024-03-31T15:53:52.792240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:56.164589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.528829+0000","last_change":"2024-03-31T15:53:47.765006+0000","last_active":"2024-03-31T15:53:53.528829+0000","last_peered":"2024-03-31T15:53:53.528829+0000","last_clean":"2024-03-31T15:53:53.528829+0000","last_became_active":"2024-03-31T15:53:47.760192+0000","last_became_peered":"2024-03-31T15:53:47.760192+0000","last_unstale":"2024-03-31T15:53:53.528829+0000","last_undegraded":"2024-03-31T15:53:53.528829+0000","last_fullsized":"2024-03-31T15:53:53.528829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:04:26.202073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790075+0000","last_change":"2024-03-31T15:53:47.765871+0000","last_active":"2024-03-31T15:53:52.790075+0000","last_peered":"2024-03-31T15:53:52.790075+0000","last_clean":"2024-03-31T15:53:52.790075+0000","last_became_active":"2024-03-31T15:53:47.764257+0000","last_became_peered":"2024-03-31T15:53:47.764257+0000","last_unstale":"2024-03-31T15:53:52.790075+0000","last_undegraded":"2024-03-31T15:53:52.790075+0000","last_fullsized":"2024-03-31T15:53:52.790075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:55:40.058921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789357+0000","last_change":"2024-03-31T15:53:47.764320+0000","last_active":"2024-03-31T15:53:52.789357+0000","last_peered":"2024-03-31T15:53:52.789357+0000","last_clean":"2024-03-31T15:53:52.789357+0000","last_became_active":"2024-03-31T15:53:47.764067+0000","last_became_peered":"2024-03-31T15:53:47.764067+0000","last_unstale":"2024-03-31T15:53:52.789357+0000","last_undegraded":"2024-03-31T15:53:52.789357+0000","last_fullsized":"2024-03-31T15:53:52.789357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.920242+0000","last_change":"2024-03-31T15:53:47.755404+0000","last_active":"2024-03-31T15:53:52.920242+0000","last_peered":"2024-03-31T15:53:52.920242+0000","last_clean":"2024-03-31T15:53:52.920242+0000","last_became_active":"2024-03-31T15:53:47.755133+0000","last_became_peered":"2024-03-31T15:53:47.755133+0000","last_unstale":"2024-03-31T15:53:52.920242+0000","last_undegraded":"2024-03-31T15:53:52.920242+0000","last_fullsized":"2024-03-31T15:53:52.920242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359051+0000","last_change":"2024-03-31T15:53:47.751494+0000","last_active":"2024-03-31T15:53:53.359051+0000","last_peered":"2024-03-31T15:53:53.359051+0000","last_clean":"2024-03-31T15:53:53.359051+0000","last_became_active":"2024-03-31T15:53:47.751219+0000","last_became_peered":"2024-03-31T15:53:47.751219+0000","last_unstale":"2024-03-31T15:53:53.359051+0000","last_undegraded":"2024-03-31T15:53:53.359051+0000","last_fullsized":"2024-03-31T15:53:53.359051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:59.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247881+0000","last_change":"2024-03-31T15:53:47.766595+0000","last_active":"2024-03-31T15:53:53.247881+0000","last_peered":"2024-03-31T15:53:53.247881+0000","last_clean":"2024-03-31T15:53:53.247881+0000","last_became_active":"2024-03-31T15:53:47.766209+0000","last_became_peered":"2024-03-31T15:53:47.766209+0000","last_unstale":"2024-03-31T15:53:53.247881+0000","last_undegraded":"2024-03-31T15:53:53.247881+0000","last_fullsized":"2024-03-31T15:53:53.247881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:05:51.207164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.793406+0000","last_change":"2024-03-31T15:53:47.762468+0000","last_active":"2024-03-31T15:53:52.793406+0000","last_peered":"2024-03-31T15:53:52.793406+0000","last_clean":"2024-03-31T15:53:52.793406+0000","last_became_active":"2024-03-31T15:53:47.762140+0000","last_became_peered":"2024-03-31T15:53:47.762140+0000","last_unstale":"2024-03-31T15:53:52.793406+0000","last_undegraded":"2024-03-31T15:53:52.793406+0000","last_fullsized":"2024-03-31T15:53:52.793406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:41:04.076153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.306895+0000","last_change":"2024-03-31T15:53:47.774515+0000","last_active":"2024-03-31T15:53:53.306895+0000","last_peered":"2024-03-31T15:53:53.306895+0000","last_clean":"2024-03-31T15:53:53.306895+0000","last_became_active":"2024-03-31T15:53:47.774203+0000","last_became_peered":"2024-03-31T15:53:47.774203+0000","last_unstale":"2024-03-31T15:53:53.306895+0000","last_undegraded":"2024-03-31T15:53:53.306895+0000","last_fullsized":"2024-03-31T15:53:53.306895+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:39:36.669797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.307016+0000","last_change":"2024-03-31T15:53:47.775151+0000","last_active":"2024-03-31T15:53:53.307016+0000","last_peered":"2024-03-31T15:53:53.307016+0000","last_clean":"2024-03-31T15:53:53.307016+0000","last_became_active":"2024-03-31T15:53:47.774661+0000","last_became_peered":"2024-03-31T15:53:47.774661+0000","last_unstale":"2024-03-31T15:53:53.307016+0000","last_undegraded":"2024-03-31T15:53:53.307016+0000","last_fullsized":"2024-03-31T15:53:53.307016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:17:22.153018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792705+0000","last_change":"2024-03-31T15:53:47.772389+0000","last_active":"2024-03-31T15:53:52.792705+0000","last_peered":"2024-03-31T15:53:52.792705+0000","last_clean":"2024-03-31T15:53:52.792705+0000","last_became_active":"2024-03-31T15:53:47.772131+0000","last_became_peered":"2024-03-31T15:53:47.772131+0000","last_unstale":"2024-03-31T15:53:52.792705+0000","last_undegraded":"2024-03-31T15:53:52.792705+0000","last_fullsized":"2024-03-31T15:53:52.792705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:56:35.224566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.247670+0000","last_change":"2024-03-31T15:53:47.772995+0000","last_active":"2024-03-31T15:53:53.247670+0000","last_peered":"2024-03-31T15:53:53.247670+0000","last_clean":"2024-03-31T15:53:53.247670+0000","last_became_active":"2024-03-31T15:53:47.772708+0000","last_became_peered":"2024-03-31T15:53:47.772708+0000","last_unstale":"2024-03-31T15:53:53.247670+0000","last_undegraded":"2024-03-31T15:53:53.247670+0000","last_fullsized":"2024-03-31T15:53:53.247670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:12:55.991650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791467+0000","last_change":"2024-03-31T15:53:47.766440+0000","last_active":"2024-03-31T15:53:52.791467+0000","last_peered":"2024-03-31T15:53:52.791467+0000","last_clean":"2024-03-31T15:53:52.791467+0000","last_became_active":"2024-03-31T15:53:47.766187+0000","last_became_peered":"2024-03-31T15:53:47.766187+0000","last_unstale":"2024-03-31T15:53:52.791467+0000","last_undegraded":"2024-03-31T15:53:52.791467+0000","last_fullsized":"2024-03-31T15:53:52.791467+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:11:53.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.790499+0000","last_change":"2024-03-31T15:53:47.778379+0000","last_active":"2024-03-31T15:53:52.790499+0000","last_peered":"2024-03-31T15:53:52.790499+0000","last_clean":"2024-03-31T15:53:52.790499+0000","last_became_active":"2024-03-31T15:53:47.778049+0000","last_became_peered":"2024-03-31T15:53:47.778049+0000","last_unstale":"2024-03-31T15:53:52.790499+0000","last_undegraded":"2024-03-31T15:53:52.790499+0000","last_fullsized":"2024-03-31T15:53:52.790499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:16:00.993238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.791659+0000","last_change":"2024-03-31T15:53:47.761184+0000","last_active":"2024-03-31T15:53:52.791659+0000","last_peered":"2024-03-31T15:53:52.791659+0000","last_clean":"2024-03-31T15:53:52.791659+0000","last_became_active":"2024-03-31T15:53:47.760863+0000","last_became_peered":"2024-03-31T15:53:47.760863+0000","last_unstale":"2024-03-31T15:53:52.791659+0000","last_undegraded":"2024-03-31T15:53:52.791659+0000","last_fullsized":"2024-03-31T15:53:52.791659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:08:38.336677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.789652+0000","last_change":"2024-03-31T15:53:47.766801+0000","last_active":"2024-03-31T15:53:52.789652+0000","last_peered":"2024-03-31T15:53:52.789652+0000","last_clean":"2024-03-31T15:53:52.789652+0000","last_became_active":"2024-03-31T15:53:47.766512+0000","last_became_peered":"2024-03-31T15:53:47.766512+0000","last_unstale":"2024-03-31T15:53:52.789652+0000","last_undegraded":"2024-03-31T15:53:52.789652+0000","last_fullsized":"2024-03-31T15:53:52.789652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:38:25.798405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:53.359475+0000","last_change":"2024-03-31T15:53:47.756732+0000","last_active":"2024-03-31T15:53:53.359475+0000","last_peered":"2024-03-31T15:53:53.359475+0000","last_clean":"2024-03-31T15:53:53.359475+0000","last_became_active":"2024-03-31T15:53:47.756466+0000","last_became_peered":"2024-03-31T15:53:47.756466+0000","last_unstale":"2024-03-31T15:53:53.359475+0000","last_undegraded":"2024-03-31T15:53:53.359475+0000","last_fullsized":"2024-03-31T15:53:53.359475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:54.147802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.921162+0000","last_change":"2024-03-31T15:53:47.763716+0000","last_active":"2024-03-31T15:53:52.921162+0000","last_peered":"2024-03-31T15:53:52.921162+0000","last_clean":"2024-03-31T15:53:52.921162+0000","last_became_active":"2024-03-31T15:53:47.763310+0000","last_became_peered":"2024-03-31T15:53:47.763310+0000","last_unstale":"2024-03-31T15:53:52.921162+0000","last_undegraded":"2024-03-31T15:53:52.921162+0000","last_fullsized":"2024-03-31T15:53:52.921162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:29:07.539833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-31T15:53:52.792492+0000","last_change":"2024-03-31T15:53:47.765680+0000","last_active":"2024-03-31T15:53:52.792492+0000","last_peered":"2024-03-31T15:53:52.792492+0000","last_clean":"2024-03-31T15:53:52.792492+0000","last_became_active":"2024-03-31T15:53:47.765516+0000","last_became_peered":"2024-03-31T15:53:47.765516+0000","last_unstale":"2024-03-31T15:53:52.792492+0000","last_undegraded":"2024-03-31T15:53:52.792492+0000","last_fullsized":"2024-03-31T15:53:52.792492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:46.727182+0000","last_clean_scrub_stamp":"2024-03-31T15:53:46.727182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:44:09.409693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509451,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":102233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":102269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509452,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":102303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509452,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27876,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343964,"statfs":{"total":96636764160,"available":96608219136,"internally_reserved":0,"allocated":864256,"data_stored":693149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":105985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":102143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":856064,"data_stored":693093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:54:18.215 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-31T15:54:18.215 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-31T15:54:18.215 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-31T15:54:18.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-31T15:54:18.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.390+0000 7f462c600640 1 Processor -- start 2024-03-31T15:54:18.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.390+0000 7f462c600640 1 -- start start 2024-03-31T15:54:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462c600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f462415e1b0 0x7f4624162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462c600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f462406c7b0 0x7f462406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462c600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f462415f780 con 0x7f462406d5f0 2024-03-31T15:54:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462c600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f46241622b0 con 0x7f462415e1b0 2024-03-31T15:54:18.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462c600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f462406da20 con 0x7f462406c7b0 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f462415e1b0 0x7f4624162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462a600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f462406d5f0 0x7f462415e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34044/0 (socket says 172.21.15.90:34044) 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f462a600640 1 -- 172.21.15.90:0/1052366089 learned_addr learned my addr 172.21.15.90:0/1052366089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629200640 1 --2- 172.21.15.90:0/1052366089 >> v2:172.21.15.102:3300/0 conn(0x7f462406c7b0 0x7f462406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 -- 172.21.15.90:0/1052366089 >> v2:172.21.15.102:3300/0 conn(0x7f462406c7b0 msgr2=0x7f462406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 --2- 172.21.15.90:0/1052366089 >> v2:172.21.15.102:3300/0 conn(0x7f462406c7b0 0x7f462406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 -- 172.21.15.90:0/1052366089 >> v1:172.21.15.90:6789/0 conn(0x7f462406d5f0 legacy=0x7f462415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 -- 172.21.15.90:0/1052366089 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462415f1c0 con 0x7f462415e1b0 2024-03-31T15:54:18.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4623e00640 1 -- 172.21.15.90:0/1052366089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296691216 0 0) 0x7f462415f780 con 0x7f462406d5f0 2024-03-31T15:54:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.394+0000 7f4629c00640 1 --2- 172.21.15.90:0/1052366089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462415e1b0 0x7f4624162740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f462406ed50 tx=0x7f461801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce94833b36188471 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4623e00640 1 -- 172.21.15.90:0/1052366089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46180b1070 con 0x7f462415e1b0 2024-03-31T15:54:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4623e00640 1 -- 172.21.15.90:0/1052366089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46180ac620 con 0x7f462415e1b0 2024-03-31T15:54:18.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4623e00640 1 -- 172.21.15.90:0/1052366089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46180ac9f0 con 0x7f462415e1b0 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- 172.21.15.90:0/1052366089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462415e1b0 msgr2=0x7f4624162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- 172.21.15.90:0/1052366089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462415e1b0 0x7f4624162740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f462406ed50 tx=0x7f461801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- 172.21.15.90:0/1052366089 shutdown_connections 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- 172.21.15.90:0/1052366089 >> v2:172.21.15.102:3300/0 conn(0x7f462406c7b0 0x7f462406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- 172.21.15.90:0/1052366089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462415e1b0 0x7f4624162740 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- 172.21.15.90:0/1052366089 >> 172.21.15.90:0/1052366089 conn(0x7f462409aa40 msgr2=0x7f462409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- 172.21.15.90:0/1052366089 shutdown_connections 2024-03-31T15:54:18.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- 172.21.15.90:0/1052366089 wait complete. 2024-03-31T15:54:18.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 Processor -- start 2024-03-31T15:54:18.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- start start 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f462406d5f0 0x7f46240af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f462415e1b0 0x7f46240af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46240d2940 con 0x7f462406c7b0 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f46240d27c0 con 0x7f462415e1b0 2024-03-31T15:54:18.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462c600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f46240d2ac0 con 0x7f462406d5f0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4629c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4629c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51648/0 (socket says 172.21.15.90:51648) 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4629c00640 1 -- 172.21.15.90:0/3176463430 learned_addr learned my addr 172.21.15.90:0/3176463430 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f462a600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f462415e1b0 0x7f46240af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.398+0000 7f4629200640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f462406d5f0 0x7f46240af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 -- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f462406d5f0 msgr2=0x7f46240af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f462406d5f0 0x7f46240af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 -- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f462415e1b0 msgr2=0x7f46240af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f462415e1b0 0x7f46240af9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 -- 172.21.15.90:0/3176463430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46240b0000 con 0x7f462406c7b0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4629c00640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f46180af040 tx=0x7f4618019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46180b9070 con 0x7f462406c7b0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46180b5410 con 0x7f462406c7b0 2024-03-31T15:54:18.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46240b02e0 con 0x7f462406c7b0 2024-03-31T15:54:18.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46180b1040 con 0x7f462406c7b0 2024-03-31T15:54:18.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46240b08a0 con 0x7f462406c7b0 2024-03-31T15:54:18.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46240686d0 con 0x7f462406c7b0 2024-03-31T15:54:18.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.402+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46180bc070 con 0x7f462406c7b0 2024-03-31T15:54:18.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.406+0000 7f4621600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 0x7f460406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.406+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4618120b90 con 0x7f462406c7b0 2024-03-31T15:54:18.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.406+0000 7f4629200640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 0x7f460406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.406+0000 7f4629200640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 0x7f460406a840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46240b1310 tx=0x7f4610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.410+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4618016070 con 0x7f462406c7b0 2024-03-31T15:54:18.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4624169f90 con 0x7f462406c7b0 2024-03-31T15:54:18.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f4621600640 1 -- 172.21.15.90:0/3176463430 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f46180f0ea0 con 0x7f462406c7b0 2024-03-31T15:54:18.596 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:18.596 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 msgr2=0x7f460406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 0x7f460406a840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46240b1310 tx=0x7f4610008040 comp rx=0 tx=0).stop 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 msgr2=0x7f46240c61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f46180af040 tx=0x7f4618019160 comp rx=0 tx=0).stop 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 shutdown_connections 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4604068380 0x7f460406a840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f462415e1b0 0x7f46240af9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f462406d5f0 0x7f46240af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 --2- 172.21.15.90:0/3176463430 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f462406c7b0 0x7f46240c61d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 >> 172.21.15.90:0/3176463430 conn(0x7f462409aa40 msgr2=0x7f462406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:18.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 shutdown_connections 2024-03-31T15:54:18.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.602+0000 7f462c600640 1 -- 172.21.15.90:0/3176463430 wait complete. 2024-03-31T15:54:18.616 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-31T15:54:18.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:18.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 Processor -- start 2024-03-31T15:54:18.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 -- start start 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9c90072840 0x7f9c90165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9015f1d0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9015f4b0 con 0x7f9c9015bfb0 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9c900cb1a0 con 0x7f9c90072840 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9c900ccaf0 0x7f9c9015e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34054/0 (socket says 172.21.15.90:34054) 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8ea00640 1 -- 172.21.15.90:0/2735198288 learned_addr learned my addr 172.21.15.90:0/2735198288 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:18.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.758+0000 7f9c8f400640 1 --2- 172.21.15.90:0/2735198288 >> v2:172.21.15.102:3300/0 conn(0x7f9c90072840 0x7f9c90165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2663832034 0 0) 0x7f9c9015f4b0 con 0x7f9c9015bfb0 2024-03-31T15:54:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c74003530 con 0x7f9c9015bfb0 2024-03-31T15:54:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098850876 0 0) 0x7f9c9015f1d0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c9015f4b0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8f400640 1 -- 172.21.15.90:0/2735198288 >> v1:172.21.15.102:6789/0 conn(0x7f9c9015bfb0 legacy=0x7f9c90072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8f400640 1 -- 172.21.15.90:0/2735198288 >> v1:172.21.15.90:6789/0 conn(0x7f9c900ccaf0 legacy=0x7f9c9015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8f400640 1 -- 172.21.15.90:0/2735198288 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9015f790 con 0x7f9c90072840 2024-03-31T15:54:18.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8f400640 1 --2- 172.21.15.90:0/2735198288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c90165c60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009d90 tx=0x7f9c8801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac96cb9b0ddad6e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880b2070 con 0x7f9c90072840 2024-03-31T15:54:18.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c880a5920 con 0x7f9c90072840 2024-03-31T15:54:18.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.762+0000 7f9c8d600640 1 -- 172.21.15.90:0/2735198288 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880a5cd0 con 0x7f9c90072840 2024-03-31T15:54:18.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2735198288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 msgr2=0x7f9c90165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2735198288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c90165c60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009d90 tx=0x7f9c8801c970 comp rx=0 tx=0).stop 2024-03-31T15:54:18.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2735198288 shutdown_connections 2024-03-31T15:54:18.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2735198288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c90165c60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2735198288 >> 172.21.15.90:0/2735198288 conn(0x7f9c900cd660 msgr2=0x7f9c900cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:18.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2735198288 shutdown_connections 2024-03-31T15:54:18.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2735198288 wait complete. 2024-03-31T15:54:18.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 Processor -- start 2024-03-31T15:54:18.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- start start 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c9014d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9c9015bfb0 0x7f9c9015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9c90070be0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9c90070a60 con 0x7f9c90072840 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9c901689c0 con 0x7f9c9015bfb0 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51652/0 (socket says 172.21.15.90:51652) 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 -- 172.21.15.90:0/2491795382 learned_addr learned my addr 172.21.15.90:0/2491795382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8ea00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c9014d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8f400640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9c9015bfb0 0x7f9c9015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 -- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9c9015bfb0 msgr2=0x7f9c9015a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9c9015bfb0 0x7f9c9015a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 -- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 msgr2=0x7f9c9014d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c9014d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.766+0000 7f9c8e000640 1 -- 172.21.15.90:0/2491795382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c900b6510 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c8e000640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00c400 tx=0x7f9c7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c017070 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c7c0052d0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c003840 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c900b67f0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c900b6db0 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c900cfb80 con 0x7f9c900ccaf0 2024-03-31T15:54:18.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c7c01a070 con 0x7f9c900ccaf0 2024-03-31T15:54:18.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 0x7f9c7006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:18.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9c7c086110 con 0x7f9c900ccaf0 2024-03-31T15:54:18.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.770+0000 7f9c8ea00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 0x7f9c7006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:18.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.774+0000 7f9c8ea00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 0x7f9c7006a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c901fa0c0 tx=0x7f9c80000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:18.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:18.774+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9c7c055d60 con 0x7f9c900ccaf0 2024-03-31T15:54:19.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.106+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c90068d00 con 0x7f9c900ccaf0 2024-03-31T15:54:19.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.106+0000 7f9c86a00640 1 -- 172.21.15.90:0/2491795382 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f9c7c05a060 con 0x7f9c900ccaf0 2024-03-31T15:54:19.102 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:19.102 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 msgr2=0x7f9c7006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 0x7f9c7006a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c901fa0c0 tx=0x7f9c80000bd0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 msgr2=0x7f9c90159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c00c400 tx=0x7f9c7c00c8c0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 shutdown_connections 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9c70068380 0x7f9c7006a840 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9c9015bfb0 0x7f9c9015a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9c900ccaf0 0x7f9c90159c30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 --2- 172.21.15.90:0/2491795382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c90072840 0x7f9c9014d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 >> 172.21.15.90:0/2491795382 conn(0x7f9c900cd660 msgr2=0x7f9c90150d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 shutdown_connections 2024-03-31T15:54:19.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.110+0000 7f9c8fe00640 1 -- 172.21.15.90:0/2491795382 wait complete. 2024-03-31T15:54:19.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:19.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:19.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 Processor -- start 2024-03-31T15:54:19.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 -- start start 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b0406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0406ce50 0x7f4b0406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b0415e8c0 con 0x7f4b040ce7a0 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4b040d0ff0 con 0x7f4b0406ce50 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b0ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4b040d12d0 con 0x7f4b0415e4e0 2024-03-31T15:54:19.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b03e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0406ce50 0x7f4b0406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b09a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4b040ce7a0 0x7f4b040cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34062/0 (socket says 172.21.15.90:34062) 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b03e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4b0406ce50 0x7f4b0406d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51668/0 (socket says 172.21.15.90:51668) 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.270+0000 7f4b09a00640 1 -- 172.21.15.90:0/2613339795 learned_addr learned my addr 172.21.15.90:0/2613339795 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b09000640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b0406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03e00640 1 -- 172.21.15.90:0/2613339795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 msgr2=0x7f4b0406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:19.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03e00640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b0406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03e00640 1 -- 172.21.15.90:0/2613339795 >> v1:172.21.15.90:6789/0 conn(0x7f4b040ce7a0 legacy=0x7f4b040cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03e00640 1 -- 172.21.15.90:0/2613339795 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b040cc710 con 0x7f4b0406ce50 2024-03-31T15:54:19.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03400640 1 -- 172.21.15.90:0/2613339795 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486810198 0 0) 0x7f4b0415e8c0 con 0x7f4b040ce7a0 2024-03-31T15:54:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03e00640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b0406d290 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097c0 tx=0x7f4af001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec293135581ceee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03400640 1 -- 172.21.15.90:0/2613339795 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00b0070 con 0x7f4b0406ce50 2024-03-31T15:54:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03400640 1 -- 172.21.15.90:0/2613339795 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af00ab600 con 0x7f4b0406ce50 2024-03-31T15:54:19.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.274+0000 7f4b03400640 1 -- 172.21.15.90:0/2613339795 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af00ab9d0 con 0x7f4b0406ce50 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2613339795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 msgr2=0x7f4b0406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b0406d290 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4af00097c0 tx=0x7f4af001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2613339795 shutdown_connections 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b0406d290 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2613339795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b0406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2613339795 >> 172.21.15.90:0/2613339795 conn(0x7f4b040d1f20 msgr2=0x7f4b040d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2613339795 shutdown_connections 2024-03-31T15:54:19.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2613339795 wait complete. 2024-03-31T15:54:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 Processor -- start 2024-03-31T15:54:19.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- start start 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b040ce7a0 0x7f4b0409d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b040ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4b040d0d90 con 0x7f4b0406ce50 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b040d0c10 con 0x7f4b040ce7a0 2024-03-31T15:54:19.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b0ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4b0416b840 con 0x7f4b0415e4e0 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51674/0 (socket says 172.21.15.90:51674) 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 -- 172.21.15.90:0/2732555164 learned_addr learned my addr 172.21.15.90:0/2732555164 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09a00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b040ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b03e00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b040ce7a0 0x7f4b0409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 -- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 msgr2=0x7f4b040ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b040ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 -- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b040ce7a0 msgr2=0x7f4b0409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b040ce7a0 0x7f4b0409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.278+0000 7f4b09000640 1 -- 172.21.15.90:0/2732555164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0415b510 con 0x7f4b0406ce50 2024-03-31T15:54:19.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b09000640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e8a0 tx=0x7f4af800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8017070 con 0x7f4b0406ce50 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af800c940 con 0x7f4b0406ce50 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0415b7f0 con 0x7f4b0406ce50 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af800ccf0 con 0x7f4b0406ce50 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b0415bdb0 con 0x7f4b0406ce50 2024-03-31T15:54:19.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b040685f0 con 0x7f4b0406ce50 2024-03-31T15:54:19.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4af801a070 con 0x7f4b0406ce50 2024-03-31T15:54:19.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 0x7f4ae406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4af8086f70 con 0x7f4b0406ce50 2024-03-31T15:54:19.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.282+0000 7f4b03e00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 0x7f4ae406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.286+0000 7f4b03e00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 0x7f4ae406a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4af00ad000 tx=0x7f4af0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.290+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4af80571e0 con 0x7f4b0406ce50 2024-03-31T15:54:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4b04073710 con 0x7f4b0406ce50 2024-03-31T15:54:19.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b00c00640 1 -- 172.21.15.90:0/2732555164 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f4af8056ae0 con 0x7f4b0406ce50 2024-03-31T15:54:19.442 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 msgr2=0x7f4ae406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 0x7f4ae406a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4af00ad000 tx=0x7f4af0008040 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 msgr2=0x7f4b04156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.446+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e8a0 tx=0x7f4af800eca0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 shutdown_connections 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4ae4068380 0x7f4ae406a840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4b0415e4e0 0x7f4b040ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b040ce7a0 0x7f4b0409d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 --2- 172.21.15.90:0/2732555164 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4b0406ce50 0x7f4b04156680 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 >> 172.21.15.90:0/2732555164 conn(0x7f4b040d1f20 msgr2=0x7f4b0415abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 shutdown_connections 2024-03-31T15:54:19.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.450+0000 7f4b0ba00640 1 -- 172.21.15.90:0/2732555164 wait complete. 2024-03-31T15:54:19.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 Processor -- start 2024-03-31T15:54:19.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 -- start start 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f97d815e4e0 0x7f97d806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d815e8c0 con 0x7f97d806ce50 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f97d80d0ff0 con 0x7f97d815e4e0 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97dd800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f97d80d12d0 con 0x7f97d80cdf20 2024-03-31T15:54:19.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97d6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f97d815e4e0 0x7f97d806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97d6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f97d815e4e0 0x7f97d806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51686/0 (socket says 172.21.15.90:51686) 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97cfe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f97d806ce50 0x7f97d80ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34072/0 (socket says 172.21.15.90:34072) 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.606+0000 7f97d6a00640 1 -- 172.21.15.90:0/733535110 learned_addr learned my addr 172.21.15.90:0/733535110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d7400640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d6a00640 1 -- 172.21.15.90:0/733535110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 msgr2=0x7f97d806d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d6a00640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d806d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d6a00640 1 -- 172.21.15.90:0/733535110 >> v1:172.21.15.90:6789/0 conn(0x7f97d806ce50 legacy=0x7f97d80ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d6000640 1 -- 172.21.15.90:0/733535110 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071005619 0 0) 0x7f97d815e8c0 con 0x7f97d806ce50 2024-03-31T15:54:19.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.610+0000 7f97d6a00640 1 -- 172.21.15.90:0/733535110 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d80cc710 con 0x7f97d815e4e0 2024-03-31T15:54:19.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97d6a00640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d806c910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97c80098a0 tx=0x7f97c801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbb88089ad25c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97d6000640 1 -- 172.21.15.90:0/733535110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c80b1070 con 0x7f97d815e4e0 2024-03-31T15:54:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97d6000640 1 -- 172.21.15.90:0/733535110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c80ac620 con 0x7f97d815e4e0 2024-03-31T15:54:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97d6000640 1 -- 172.21.15.90:0/733535110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c80ac9f0 con 0x7f97d815e4e0 2024-03-31T15:54:19.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- 172.21.15.90:0/733535110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 msgr2=0x7f97d806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d806c910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f97c80098a0 tx=0x7f97c801c4b0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- 172.21.15.90:0/733535110 shutdown_connections 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 --2- 172.21.15.90:0/733535110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d806c910 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- 172.21.15.90:0/733535110 >> 172.21.15.90:0/733535110 conn(0x7f97d80d1f20 msgr2=0x7f97d80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- 172.21.15.90:0/733535110 shutdown_connections 2024-03-31T15:54:19.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- 172.21.15.90:0/733535110 wait complete. 2024-03-31T15:54:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 Processor -- start 2024-03-31T15:54:19.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.614+0000 7f97dd800640 1 -- start start 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97d806ce50 0x7f97d8156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d80c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97d816b1b0 con 0x7f97d815e4e0 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97d816b030 con 0x7f97d806ce50 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f97d816b330 con 0x7f97d80cdf20 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51694/0 (socket says 172.21.15.90:51694) 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 -- 172.21.15.90:0/665827214 learned_addr learned my addr 172.21.15.90:0/665827214 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97cfe00640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d80c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d6a00640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97d806ce50 0x7f97d8156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 -- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 msgr2=0x7f97d80c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d80c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 -- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97d806ce50 msgr2=0x7f97d8156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97d806ce50 0x7f97d8156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 -- 172.21.15.90:0/665827214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d80cc060 con 0x7f97d815e4e0 2024-03-31T15:54:19.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97d7400640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97d000efc0 tx=0x7f97d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0018020 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d0014480 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0014870 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d80c7aa0 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d80c7d40 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.618+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d80685f0 con 0x7f97d815e4e0 2024-03-31T15:54:19.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.622+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f97d001b020 con 0x7f97d815e4e0 2024-03-31T15:54:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.622+0000 7f97cea00640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 0x7f97b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.622+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f97d0087ad0 con 0x7f97d815e4e0 2024-03-31T15:54:19.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.622+0000 7f97d6a00640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 0x7f97b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.622+0000 7f97d6a00640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 0x7f97b406a840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f97c80ae000 tx=0x7f97c80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.630+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f97d0022030 con 0x7f97d815e4e0 2024-03-31T15:54:19.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97d815aea0 con 0x7f97d815e4e0 2024-03-31T15:54:19.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97cea00640 1 -- 172.21.15.90:0/665827214 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f97d0010040 con 0x7f97d815e4e0 2024-03-31T15:54:19.789 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 msgr2=0x7f97b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 0x7f97b406a840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f97c80ae000 tx=0x7f97c80a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 msgr2=0x7f97d80b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97d000efc0 tx=0x7f97d000c490 comp rx=0 tx=0).stop 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 shutdown_connections 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97b4068380 0x7f97b406a840 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97d815e4e0 0x7f97d80b5c80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97d80cdf20 0x7f97d80c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 --2- 172.21.15.90:0/665827214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97d806ce50 0x7f97d8156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 >> 172.21.15.90:0/665827214 conn(0x7f97d80d1f20 msgr2=0x7f97d815a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 shutdown_connections 2024-03-31T15:54:19.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.794+0000 7f97dd800640 1 -- 172.21.15.90:0/665827214 wait complete. 2024-03-31T15:54:19.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.954+0000 7fd3e7200640 1 Processor -- start 2024-03-31T15:54:19.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.954+0000 7fd3e7200640 1 -- start start 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3e815bf70 0x7fd3e80cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3e815c350 0x7fd3e806b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e7200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e80b6290 con 0x7fd3e80af8d0 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e80b5e00 con 0x7fd3e815bf70 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e80b51f0 con 0x7fd3e815c350 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3e815bf70 0x7fd3e80cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd3e815bf70 0x7fd3e80cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51700/0 (socket says 172.21.15.90:51700) 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 -- 172.21.15.90:0/601658960 learned_addr learned my addr 172.21.15.90:0/601658960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3dfe00640 1 --2- 172.21.15.90:0/601658960 >> v2:172.21.15.102:3300/0 conn(0x7fd3e815c350 0x7fd3e806b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 -- 172.21.15.90:0/601658960 >> v2:172.21.15.102:3300/0 conn(0x7fd3e815c350 msgr2=0x7fd3e806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 --2- 172.21.15.90:0/601658960 >> v2:172.21.15.102:3300/0 conn(0x7fd3e815c350 0x7fd3e806b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 -- 172.21.15.90:0/601658960 >> v1:172.21.15.90:6789/0 conn(0x7fd3e80af8d0 legacy=0x7fd3e815fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.958+0000 7fd3e5e00640 1 -- 172.21.15.90:0/601658960 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e80b54d0 con 0x7fd3e815bf70 2024-03-31T15:54:19.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e5e00640 1 --2- 172.21.15.90:0/601658960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815bf70 0x7fd3e80cd660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80cdf00 tx=0x7fd3d801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113ff3c8b63c64cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e5400640 1 -- 172.21.15.90:0/601658960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80ab600 con 0x7fd3e815bf70 2024-03-31T15:54:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e5400640 1 -- 172.21.15.90:0/601658960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d80b0070 con 0x7fd3e815bf70 2024-03-31T15:54:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e5400640 1 -- 172.21.15.90:0/601658960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80ab9c0 con 0x7fd3e815bf70 2024-03-31T15:54:19.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- 172.21.15.90:0/601658960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815bf70 msgr2=0x7fd3e80cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- 172.21.15.90:0/601658960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815bf70 0x7fd3e80cd660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80cdf00 tx=0x7fd3d801c4f0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- 172.21.15.90:0/601658960 shutdown_connections 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- 172.21.15.90:0/601658960 >> v2:172.21.15.102:3300/0 conn(0x7fd3e815c350 0x7fd3e806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- 172.21.15.90:0/601658960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815bf70 0x7fd3e80cd660 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- 172.21.15.90:0/601658960 >> 172.21.15.90:0/601658960 conn(0x7fd3e80b1f60 msgr2=0x7fd3e80b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- 172.21.15.90:0/601658960 shutdown_connections 2024-03-31T15:54:19.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- 172.21.15.90:0/601658960 wait complete. 2024-03-31T15:54:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 Processor -- start 2024-03-31T15:54:19.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- start start 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 0x7fd3e8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3e815bf70 0x7fd3e815e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e8169120 con 0x7fd3e815c350 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e8168fa0 con 0x7fd3e80af8d0 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.962+0000 7fd3e7200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e81692a0 con 0x7fd3e815bf70 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 0x7fd3e8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 0x7fd3e8157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47386/0 (socket says 172.21.15.90:47386) 2024-03-31T15:54:19.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51710/0 (socket says 172.21.15.90:51710) 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e5e00640 1 -- 172.21.15.90:0/1645563872 learned_addr learned my addr 172.21.15.90:0/1645563872 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3dfe00640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3e815bf70 0x7fd3e815e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 -- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3e815bf70 msgr2=0x7fd3e815e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3e815bf70 0x7fd3e815e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 -- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 msgr2=0x7fd3e8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 0x7fd3e8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 -- 172.21.15.90:0/1645563872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e806b960 con 0x7fd3e815c350 2024-03-31T15:54:19.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e6800640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8071440 tx=0x7fd3e000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0003b80 con 0x7fd3e815c350 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e0003d20 con 0x7fd3e815c350 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0005310 con 0x7fd3e815c350 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e815f7b0 con 0x7fd3e815c350 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e815ef10 con 0x7fd3e815c350 2024-03-31T15:54:19.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.966+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e8068d00 con 0x7fd3e815c350 2024-03-31T15:54:19.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.970+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd3e00056c0 con 0x7fd3e815c350 2024-03-31T15:54:19.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.970+0000 7fd3de000640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 0x7fd3c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:19.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.970+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd3e0086c90 con 0x7fd3e815c350 2024-03-31T15:54:19.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.970+0000 7fd3e5e00640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 0x7fd3c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:19.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.970+0000 7fd3e5e00640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 0x7fd3c406a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd3d801c530 tx=0x7fd3d80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:19.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:19.974+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd3e0056820 con 0x7fd3e815c350 2024-03-31T15:54:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.130+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd3e80af570 con 0x7fd3e815c350 2024-03-31T15:54:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3de000640 1 -- 172.21.15.90:0/1645563872 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fd3e005ab20 con 0x7fd3e815c350 2024-03-31T15:54:20.128 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 msgr2=0x7fd3c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 0x7fd3c406a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd3d801c530 tx=0x7fd3d80a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 msgr2=0x7fd3e80997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8071440 tx=0x7fd3e000cb90 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 shutdown_connections 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd3c4068380 0x7fd3c406a840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e815c350 0x7fd3e80997a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd3e815bf70 0x7fd3e815e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 --2- 172.21.15.90:0/1645563872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e80af8d0 0x7fd3e8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 >> 172.21.15.90:0/1645563872 conn(0x7fd3e80b1f60 msgr2=0x7fd3e80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 shutdown_connections 2024-03-31T15:54:20.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.134+0000 7fd3e7200640 1 -- 172.21.15.90:0/1645563872 wait complete. 2024-03-31T15:54:20.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:20.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 Processor -- start 2024-03-31T15:54:20.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 -- start start 2024-03-31T15:54:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd22406d460 0x7fd2240cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd22415e8c0 con 0x7fd22415e4e0 2024-03-31T15:54:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd2240d0ff0 con 0x7fd22406d460 2024-03-31T15:54:20.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd22ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd2240d12d0 con 0x7fd22406d020 2024-03-31T15:54:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd223e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd22406d460 0x7fd2240cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd22406d460 0x7fd2240cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:47390/0 (socket says 172.21.15.90:47390) 2024-03-31T15:54:20.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 -- 172.21.15.90:0/1377899675 learned_addr learned my addr 172.21.15.90:0/1377899675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd223e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50380/0 (socket says 172.21.15.90:50380) 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 -- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 msgr2=0x7fd2240ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 -- 172.21.15.90:0/1377899675 >> v1:172.21.15.102:6789/0 conn(0x7fd22415e4e0 legacy=0x7fd22406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd229a00640 1 -- 172.21.15.90:0/1377899675 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2240cc710 con 0x7fd22406d460 2024-03-31T15:54:20.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.294+0000 7fd223e00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd229a00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22406d460 0x7fd2240cd450 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd21c009e50 tx=0x7fd21c01c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=129b93137d3bd2e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd223400640 1 -- 172.21.15.90:0/1377899675 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0a5660 con 0x7fd22406d460 2024-03-31T15:54:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd223400640 1 -- 172.21.15.90:0/1377899675 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21c0a5800 con 0x7fd22406d460 2024-03-31T15:54:20.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd223400640 1 -- 172.21.15.90:0/1377899675 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0a5bd0 con 0x7fd22406d460 2024-03-31T15:54:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 -- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22406d460 msgr2=0x7fd2240cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22406d460 0x7fd2240cd450 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd21c009e50 tx=0x7fd21c01c9a0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 -- 172.21.15.90:0/1377899675 shutdown_connections 2024-03-31T15:54:20.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22406d460 0x7fd2240cd450 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 --2- 172.21.15.90:0/1377899675 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 -- 172.21.15.90:0/1377899675 >> 172.21.15.90:0/1377899675 conn(0x7fd2240d1f20 msgr2=0x7fd2240d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 -- 172.21.15.90:0/1377899675 shutdown_connections 2024-03-31T15:54:20.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.298+0000 7fd22ba00640 1 -- 172.21.15.90:0/1377899675 wait complete. 2024-03-31T15:54:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 Processor -- start 2024-03-31T15:54:20.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- start start 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22415e4e0 0x7fd224144d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240d0d90 con 0x7fd22406d460 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd2240d0c10 con 0x7fd22415e4e0 2024-03-31T15:54:20.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd22416b030 con 0x7fd22406d020 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51720/0 (socket says 172.21.15.90:51720) 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 -- 172.21.15.90:0/3417869081 learned_addr learned my addr 172.21.15.90:0/3417869081 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd229a00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22415e4e0 0x7fd224144d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd229000640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 -- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 msgr2=0x7fd2240cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 -- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22415e4e0 msgr2=0x7fd224144d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22415e4e0 0x7fd224144d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 -- 172.21.15.90:0/3417869081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2240b3e80 con 0x7fd22406d460 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd223e00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd21000ebe0 tx=0x7fd21000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210017070 con 0x7fd22406d460 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd21000cc30 con 0x7fd22406d460 2024-03-31T15:54:20.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210013470 con 0x7fd22406d460 2024-03-31T15:54:20.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2240b4100 con 0x7fd22406d460 2024-03-31T15:54:20.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2240b46c0 con 0x7fd22406d460 2024-03-31T15:54:20.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.302+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2240685f0 con 0x7fd22406d460 2024-03-31T15:54:20.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.306+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd21001a070 con 0x7fd22406d460 2024-03-31T15:54:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.306+0000 7fd220c00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 0x7fd20406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.306+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd210087280 con 0x7fd22406d460 2024-03-31T15:54:20.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.306+0000 7fd229000640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 0x7fd20406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.306+0000 7fd229000640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 0x7fd20406a840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2180015b0 tx=0x7fd218009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.310+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd2240b3e80 con 0x7fd22406d460 2024-03-31T15:54:20.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.466+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd2240cc2e0 con 0x7fd22406d460 2024-03-31T15:54:20.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.466+0000 7fd220c00640 1 -- 172.21.15.90:0/3417869081 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fd210057020 con 0x7fd22406d460 2024-03-31T15:54:20.462 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:20.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 msgr2=0x7fd20406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 0x7fd20406a840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2180015b0 tx=0x7fd218009040 comp rx=0 tx=0).stop 2024-03-31T15:54:20.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 msgr2=0x7fd2240b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd21000ebe0 tx=0x7fd21000c520 comp rx=0 tx=0).stop 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 shutdown_connections 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd204068380 0x7fd20406a840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd22415e4e0 0x7fd224144d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd22406d460 0x7fd2240b3940 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 --2- 172.21.15.90:0/3417869081 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd22406d020 0x7fd2240cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 >> 172.21.15.90:0/3417869081 conn(0x7fd2240d1f20 msgr2=0x7fd22415d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 shutdown_connections 2024-03-31T15:54:20.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.470+0000 7fd22ba00640 1 -- 172.21.15.90:0/3417869081 wait complete. 2024-03-31T15:54:20.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:54:20.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 Processor -- start 2024-03-31T15:54:20.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 -- start start 2024-03-31T15:54:20.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f363c15e4c0 0x7f363c0d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363c15e8a0 con 0x7f363c0d4700 2024-03-31T15:54:20.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f363c06cf90 con 0x7f363c06c990 2024-03-31T15:54:20.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3640a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f363c06d270 con 0x7f363c15e4c0 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f363c0d4700 0x7f363c06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34090/0 (socket says 172.21.15.90:34090) 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639400640 1 -- 172.21.15.90:0/45510140 learned_addr learned my addr 172.21.15.90:0/45510140 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639e00640 1 --2- 172.21.15.90:0/45510140 >> v2:172.21.15.102:3300/0 conn(0x7f363c15e4c0 0x7f363c0d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598647342 0 0) 0x7f363c15e8a0 con 0x7f363c0d4700 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3620003530 con 0x7f363c0d4700 2024-03-31T15:54:20.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639e00640 1 -- 172.21.15.90:0/45510140 >> v1:172.21.15.102:6789/0 conn(0x7f363c06c990 legacy=0x7f363c0cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2019896183 0 0) 0x7f363c06cf90 con 0x7f363c06c990 2024-03-31T15:54:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639e00640 1 -- 172.21.15.90:0/45510140 >> v1:172.21.15.90:6789/0 conn(0x7f363c0d4700 legacy=0x7f363c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.694+0000 7f3639e00640 1 -- 172.21.15.90:0/45510140 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c06d550 con 0x7f363c15e4c0 2024-03-31T15:54:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3639e00640 1 --2- 172.21.15.90:0/45510140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c0d41c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f362c0097c0 tx=0x7f362c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=926bfdc69616509d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0b0070 con 0x7f363c15e4c0 2024-03-31T15:54:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362c0ab600 con 0x7f363c15e4c0 2024-03-31T15:54:20.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3638a00640 1 -- 172.21.15.90:0/45510140 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0ab9d0 con 0x7f363c15e4c0 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 -- 172.21.15.90:0/45510140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 msgr2=0x7f363c0d41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 --2- 172.21.15.90:0/45510140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c0d41c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f362c0097c0 tx=0x7f362c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 -- 172.21.15.90:0/45510140 shutdown_connections 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 --2- 172.21.15.90:0/45510140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c0d41c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 -- 172.21.15.90:0/45510140 >> 172.21.15.90:0/45510140 conn(0x7f363c08db60 msgr2=0x7f363c09d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 -- 172.21.15.90:0/45510140 shutdown_connections 2024-03-31T15:54:20.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.698+0000 7f3640a00640 1 -- 172.21.15.90:0/45510140 wait complete. 2024-03-31T15:54:20.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 Processor -- start 2024-03-31T15:54:20.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- start start 2024-03-31T15:54:20.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 0x7f363c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f363c0d0770 con 0x7f363c0d4700 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f363c0d05f0 con 0x7f363c15e4c0 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f363c16c000 con 0x7f363c06c990 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51736/0 (socket says 172.21.15.90:51736) 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639400640 1 -- 172.21.15.90:0/472047965 learned_addr learned my addr 172.21.15.90:0/472047965 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 0x7f363c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f363a800640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 -- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 msgr2=0x7f363c14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c14c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 -- 172.21.15.90:0/472047965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 msgr2=0x7f363c1517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 -- 172.21.15.90:0/472047965 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c150290 con 0x7f363c06c990 2024-03-31T15:54:20.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639400640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:20.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3639e00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 0x7f363c0cd480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f362c0ab000 tx=0x7f362c019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0b9070 con 0x7f363c06c990 2024-03-31T15:54:20.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f362c0b5480 con 0x7f363c06c990 2024-03-31T15:54:20.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362c0b0040 con 0x7f363c06c990 2024-03-31T15:54:20.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363c0cdb10 con 0x7f363c06c990 2024-03-31T15:54:20.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f363c150810 con 0x7f363c06c990 2024-03-31T15:54:20.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.702+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363c0686d0 con 0x7f363c06c990 2024-03-31T15:54:20.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.706+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f362c0bc070 con 0x7f363c06c990 2024-03-31T15:54:20.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.706+0000 7f3632000640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 0x7f361c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:20.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.706+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f362c120b10 con 0x7f363c06c990 2024-03-31T15:54:20.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.706+0000 7f3639400640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 0x7f361c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:20.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.706+0000 7f3639400640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 0x7f361c06a870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f363c151700 tx=0x7f3628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:20.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.710+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f362c0c3030 con 0x7f363c06c990 2024-03-31T15:54:20.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.990+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f363c0d1a90 con 0x7f363c06c990 2024-03-31T15:54:20.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3632000640 1 -- 172.21.15.90:0/472047965 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f362c016090 con 0x7f363c06c990 2024-03-31T15:54:20.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:20.988 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:54:20.988 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:54:20.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 msgr2=0x7f361c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 0x7f361c06a870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f363c151700 tx=0x7f3628009040 comp rx=0 tx=0).stop 2024-03-31T15:54:20.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 msgr2=0x7f363c0cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:20.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 0x7f363c0cd480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f362c0ab000 tx=0x7f362c019240 comp rx=0 tx=0).stop 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 shutdown_connections 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f361c0683b0 0x7f361c06a870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c15e4c0 0x7f363c14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f363c0d4700 0x7f363c1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 --2- 172.21.15.90:0/472047965 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f363c06c990 0x7f363c0cd480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 >> 172.21.15.90:0/472047965 conn(0x7f363c08db60 msgr2=0x7f363c0b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 shutdown_connections 2024-03-31T15:54:20.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:20.994+0000 7f3640a00640 1 -- 172.21.15.90:0/472047965 wait complete. 2024-03-31T15:54:21.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:21.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 Processor -- start 2024-03-31T15:54:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 -- start start 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f119006d7c0 0x7f119015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119015f780 con 0x7f119006c7b0 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f119015f1c0 con 0x7f119015e1b0 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f1195400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f11901622b0 con 0x7f119006d7c0 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118de00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f119006c7b0 0x7f119006d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34098/0 (socket says 172.21.15.90:34098) 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118de00640 1 -- 172.21.15.90:0/2797165851 learned_addr learned my addr 172.21.15.90:0/2797165851 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118f200640 1 --2- 172.21.15.90:0/2797165851 >> v2:172.21.15.102:3300/0 conn(0x7f119006d7c0 0x7f119015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118d400640 1 -- 172.21.15.90:0/2797165851 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1642911244 0 0) 0x7f119015f780 con 0x7f119006c7b0 2024-03-31T15:54:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118d400640 1 -- 172.21.15.90:0/2797165851 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1174003530 con 0x7f119006c7b0 2024-03-31T15:54:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118f200640 1 -- 172.21.15.90:0/2797165851 >> v1:172.21.15.102:6789/0 conn(0x7f119015e1b0 legacy=0x7f1190162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118f200640 1 -- 172.21.15.90:0/2797165851 >> v1:172.21.15.90:6789/0 conn(0x7f119006c7b0 legacy=0x7f119006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.150+0000 7f118f200640 1 -- 172.21.15.90:0/2797165851 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119015f4a0 con 0x7f119006d7c0 2024-03-31T15:54:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f118f200640 1 --2- 172.21.15.90:0/2797165851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f119015e590 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1188009d90 tx=0x7f118801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f28e1506d1d873f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f118d400640 1 -- 172.21.15.90:0/2797165851 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11880b2070 con 0x7f119006d7c0 2024-03-31T15:54:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f118d400640 1 -- 172.21.15.90:0/2797165851 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11880a5970 con 0x7f119006d7c0 2024-03-31T15:54:21.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f118d400640 1 -- 172.21.15.90:0/2797165851 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11880a5d20 con 0x7f119006d7c0 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 -- 172.21.15.90:0/2797165851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 msgr2=0x7f119015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 --2- 172.21.15.90:0/2797165851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f119015e590 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1188009d90 tx=0x7f118801c8b0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 -- 172.21.15.90:0/2797165851 shutdown_connections 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 --2- 172.21.15.90:0/2797165851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f119015e590 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 -- 172.21.15.90:0/2797165851 >> 172.21.15.90:0/2797165851 conn(0x7f119009aa40 msgr2=0x7f119009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:21.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 -- 172.21.15.90:0/2797165851 shutdown_connections 2024-03-31T15:54:21.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.154+0000 7f1195400640 1 -- 172.21.15.90:0/2797165851 wait complete. 2024-03-31T15:54:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.158+0000 7f1195400640 1 Processor -- start 2024-03-31T15:54:21.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.158+0000 7f1195400640 1 -- start start 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f1190148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 0x7f119015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11900ce7a0 con 0x7f119006c7b0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11900d2a50 con 0x7f119006d7c0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f11900ce920 con 0x7f119015e1b0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51746/0 (socket says 172.21.15.90:51746) 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 -- 172.21.15.90:0/3055069603 learned_addr learned my addr 172.21.15.90:0/3055069603 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118de00640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f1190148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118f200640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 0x7f119015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 -- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 msgr2=0x7f119015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 0x7f119015af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 -- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 msgr2=0x7f1190148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f1190148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 -- 172.21.15.90:0/3055069603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119015d3a0 con 0x7f119006c7b0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118f200640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 0x7f119015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f118e800640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f118000c3a0 tx=0x7f118000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11800052d0 con 0x7f119006c7b0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1180005470 con 0x7f119006c7b0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1180007260 con 0x7f119006c7b0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119015d680 con 0x7f119006c7b0 2024-03-31T15:54:21.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1190148690 con 0x7f119006c7b0 2024-03-31T15:54:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11900686d0 con 0x7f119006c7b0 2024-03-31T15:54:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.162+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1180007400 con 0x7f119006c7b0 2024-03-31T15:54:21.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.166+0000 7f1186a00640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 0x7f117006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.166+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f11800860d0 con 0x7f119006c7b0 2024-03-31T15:54:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.166+0000 7f118de00640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 0x7f117006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.166+0000 7f118de00640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 0x7f117006a840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f11900c8d80 tx=0x7f117c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.170+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f11800563e0 con 0x7f119006c7b0 2024-03-31T15:54:21.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f119015e590 con 0x7f119006c7b0 2024-03-31T15:54:21.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1186a00640 1 -- 172.21.15.90:0/3055069603 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f1180055ce0 con 0x7f119006c7b0 2024-03-31T15:54:21.325 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:21.326 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 msgr2=0x7f117006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 0x7f117006a840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f11900c8d80 tx=0x7f117c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 msgr2=0x7f11900c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f118000c3a0 tx=0x7f118000c860 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 shutdown_connections 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1170068380 0x7f117006a840 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f119015e1b0 0x7f119015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.330+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f119006d7c0 0x7f1190148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.334+0000 7f1195400640 1 --2- 172.21.15.90:0/3055069603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f119006c7b0 0x7f11900c8ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.334+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 >> 172.21.15.90:0/3055069603 conn(0x7f119009aa40 msgr2=0x7f119006f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.334+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 shutdown_connections 2024-03-31T15:54:21.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.334+0000 7f1195400640 1 -- 172.21.15.90:0/3055069603 wait complete. 2024-03-31T15:54:21.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:21.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 Processor -- start 2024-03-31T15:54:21.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 -- start start 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a3015e3e0 0x7f7a30161e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300cbb70 0x7f7a300cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a300d0b50 con 0x7f7a300706c0 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7a30162db0 con 0x7f7a3015e3e0 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a35800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7a30162510 con 0x7f7a300cbb70 2024-03-31T15:54:21.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a3015e3e0 0x7f7a30161e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a3015e3e0 0x7f7a30161e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51756/0 (socket says 172.21.15.90:51756) 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 -- 172.21.15.90:0/89635317 learned_addr learned my addr 172.21.15.90:0/89635317 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2e000640 1 --1- 172.21.15.90:0/89635317 >> v1:172.21.15.90:6789/0 conn(0x7f7a300706c0 0x7f7a300cb460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34104/0 (socket says 172.21.15.90:34104) 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2f400640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300cbb70 0x7f7a300cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 -- 172.21.15.90:0/89635317 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300cbb70 msgr2=0x7f7a300cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300cbb70 0x7f7a300cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 -- 172.21.15.90:0/89635317 >> v1:172.21.15.90:6789/0 conn(0x7f7a300706c0 legacy=0x7f7a300cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.490+0000 7f7a2ea00640 1 -- 172.21.15.90:0/89635317 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a301627f0 con 0x7f7a3015e3e0 2024-03-31T15:54:21.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a3015e3e0 0x7f7a30161e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009870 tx=0x7f7a2001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e24aa63d9f9277 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a2d600640 1 -- 172.21.15.90:0/89635317 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a200aa430 con 0x7f7a3015e3e0 2024-03-31T15:54:21.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a2d600640 1 -- 172.21.15.90:0/89635317 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a200af070 con 0x7f7a3015e3e0 2024-03-31T15:54:21.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a2d600640 1 -- 172.21.15.90:0/89635317 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a200aa7f0 con 0x7f7a3015e3e0 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- 172.21.15.90:0/89635317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a3015e3e0 msgr2=0x7f7a30161e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a3015e3e0 0x7f7a30161e10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009870 tx=0x7f7a2001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- 172.21.15.90:0/89635317 shutdown_connections 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300cbb70 0x7f7a300cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 --2- 172.21.15.90:0/89635317 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a3015e3e0 0x7f7a30161e10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- 172.21.15.90:0/89635317 >> 172.21.15.90:0/89635317 conn(0x7f7a300cde90 msgr2=0x7f7a300ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- 172.21.15.90:0/89635317 shutdown_connections 2024-03-31T15:54:21.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- 172.21.15.90:0/89635317 wait complete. 2024-03-31T15:54:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 Processor -- start 2024-03-31T15:54:21.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.494+0000 7f7a35800640 1 -- start start 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 0x7f7a301589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 0x7f7a3015f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300d0110 con 0x7f7a300cbb70 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300cff90 con 0x7f7a3015e3e0 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7a300d0290 con 0x7f7a300706c0 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51772/0 (socket says 172.21.15.90:51772) 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 -- 172.21.15.90:0/1178577391 learned_addr learned my addr 172.21.15.90:0/1178577391 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 0x7f7a301589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2f400640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 0x7f7a3015f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 -- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 msgr2=0x7f7a301589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 0x7f7a301589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 -- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 msgr2=0x7f7a3015f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 0x7f7a3015f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 -- 172.21.15.90:0/1178577391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a300c9140 con 0x7f7a300cbb70 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2f400640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 0x7f7a3015f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 0x7f7a301589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a2e000640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00c370 tx=0x7f7a1c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c0074e0 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1c007680 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c004a30 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a300c9420 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a300c99e0 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.498+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a300685f0 con 0x7f7a300cbb70 2024-03-31T15:54:21.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.502+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7a1c01a020 con 0x7f7a300cbb70 2024-03-31T15:54:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.502+0000 7f7a26a00640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 0x7f7a1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.502+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7a1c0864f0 con 0x7f7a300cbb70 2024-03-31T15:54:21.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.502+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 0x7f7a1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.502+0000 7f7a2ea00640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 0x7f7a1006a840 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a200ae000 tx=0x7f7a200a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.506+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7a1c056760 con 0x7f7a300cbb70 2024-03-31T15:54:21.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7a300731a0 con 0x7f7a300cbb70 2024-03-31T15:54:21.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a26a00640 1 -- 172.21.15.90:0/1178577391 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f7a1c056060 con 0x7f7a300cbb70 2024-03-31T15:54:21.661 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 msgr2=0x7f7a1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 0x7f7a1006a840 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a200ae000 tx=0x7f7a200a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 msgr2=0x7f7a300ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00c370 tx=0x7f7a1c00c830 comp rx=0 tx=0).stop 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 shutdown_connections 2024-03-31T15:54:21.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a10068380 0x7f7a1006a840 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a3015e3e0 0x7f7a3015f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a300cbb70 0x7f7a300ca1e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 --2- 172.21.15.90:0/1178577391 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a300706c0 0x7f7a301589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 >> 172.21.15.90:0/1178577391 conn(0x7f7a300cde90 msgr2=0x7f7a300d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 shutdown_connections 2024-03-31T15:54:21.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.666+0000 7f7a35800640 1 -- 172.21.15.90:0/1178577391 wait complete. 2024-03-31T15:54:21.683 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:21.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.826+0000 7fce79e00640 1 Processor -- start 2024-03-31T15:54:21.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.826+0000 7fce79e00640 1 -- start start 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce79e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fce7406d020 0x7fce740ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce79e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7415e8c0 con 0x7fce7415e4e0 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce79e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce740d0ff0 con 0x7fce7406d490 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce79e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fce740d12d0 con 0x7fce7406d020 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fce7415e4e0 0x7fce7406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34118/0 (socket says 172.21.15.90:34118) 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72a00640 1 -- 172.21.15.90:0/353604336 learned_addr learned my addr 172.21.15.90:0/353604336 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:21.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72000640 1 --2- 172.21.15.90:0/353604336 >> v2:172.21.15.102:3300/0 conn(0x7fce7406d020 0x7fce740ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce71600640 1 -- 172.21.15.90:0/353604336 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269564388 0 0) 0x7fce7415e8c0 con 0x7fce7415e4e0 2024-03-31T15:54:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce71600640 1 -- 172.21.15.90:0/353604336 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce58003530 con 0x7fce7415e4e0 2024-03-31T15:54:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72000640 1 -- 172.21.15.90:0/353604336 >> v1:172.21.15.102:6789/0 conn(0x7fce7406d490 legacy=0x7fce740cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72000640 1 -- 172.21.15.90:0/353604336 >> v1:172.21.15.90:6789/0 conn(0x7fce7415e4e0 legacy=0x7fce7406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.830+0000 7fce72000640 1 -- 172.21.15.90:0/353604336 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce740cc710 con 0x7fce7406d020 2024-03-31T15:54:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce72000640 1 --2- 172.21.15.90:0/353604336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7406d020 0x7fce740ccf10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce600097f0 tx=0x7fce6001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3d67255f8860fc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce71600640 1 -- 172.21.15.90:0/353604336 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce600b0070 con 0x7fce7406d020 2024-03-31T15:54:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce71600640 1 -- 172.21.15.90:0/353604336 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce600ab600 con 0x7fce7406d020 2024-03-31T15:54:21.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce71600640 1 -- 172.21.15.90:0/353604336 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce600ab9d0 con 0x7fce7406d020 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- 172.21.15.90:0/353604336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7406d020 msgr2=0x7fce740ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 --2- 172.21.15.90:0/353604336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7406d020 0x7fce740ccf10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce600097f0 tx=0x7fce6001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- 172.21.15.90:0/353604336 shutdown_connections 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 --2- 172.21.15.90:0/353604336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7406d020 0x7fce740ccf10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- 172.21.15.90:0/353604336 >> 172.21.15.90:0/353604336 conn(0x7fce740d1f20 msgr2=0x7fce740d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- 172.21.15.90:0/353604336 shutdown_connections 2024-03-31T15:54:21.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- 172.21.15.90:0/353604336 wait complete. 2024-03-31T15:54:21.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 Processor -- start 2024-03-31T15:54:21.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 -- start start 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce7406d490 0x7fce74157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.834+0000 7fce79e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7415e4e0 0x7fce741fa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce740d0d90 con 0x7fce7406d020 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fce740d0c10 con 0x7fce7415e4e0 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fce7416c040 con 0x7fce7406d490 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51786/0 (socket says 172.21.15.90:51786) 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 -- 172.21.15.90:0/3600873889 learned_addr learned my addr 172.21.15.90:0/3600873889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72000640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce7406d490 0x7fce74157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce73400640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7415e4e0 0x7fce741fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 -- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce7406d490 msgr2=0x7fce74157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce7406d490 0x7fce74157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 -- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7415e4e0 msgr2=0x7fce741fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7415e4e0 0x7fce741fa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 -- 172.21.15.90:0/3600873889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce74158690 con 0x7fce7406d020 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce72a00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c500 tx=0x7fce6400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64017070 con 0x7fce7406d020 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce64005cf0 con 0x7fce7406d020 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce640072f0 con 0x7fce7406d020 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce74158970 con 0x7fce7406d020 2024-03-31T15:54:21.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce74158f30 con 0x7fce7406d020 2024-03-31T15:54:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce740685f0 con 0x7fce7406d020 2024-03-31T15:54:21.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.838+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce6401a070 con 0x7fce7406d020 2024-03-31T15:54:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.842+0000 7fce6b400640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 0x7fce5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.842+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fce64086020 con 0x7fce7406d020 2024-03-31T15:54:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.842+0000 7fce72000640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 0x7fce5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:21.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.842+0000 7fce72000640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 0x7fce5406a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fce600ad000 tx=0x7fce60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:21.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:21.846+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fce64055ce0 con 0x7fce7406d020 2024-03-31T15:54:21.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.002+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fce740ca040 con 0x7fce7406d020 2024-03-31T15:54:21.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce6b400640 1 -- 172.21.15.90:0/3600873889 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fce64059fe0 con 0x7fce7406d020 2024-03-31T15:54:22.000 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 msgr2=0x7fce5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 0x7fce5406a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fce600ad000 tx=0x7fce60008040 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 msgr2=0x7fce741578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fce6400c500 tx=0x7fce6400c9c0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 shutdown_connections 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fce54068380 0x7fce5406a840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce7415e4e0 0x7fce741fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce7406d490 0x7fce74157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 --2- 172.21.15.90:0/3600873889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce7406d020 0x7fce741578f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 >> 172.21.15.90:0/3600873889 conn(0x7fce740d1f20 msgr2=0x7fce74155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 shutdown_connections 2024-03-31T15:54:22.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.006+0000 7fce79e00640 1 -- 172.21.15.90:0/3600873889 wait complete. 2024-03-31T15:54:22.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:22.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.170+0000 7fe5f4a00640 1 Processor -- start 2024-03-31T15:54:22.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.170+0000 7fe5f4a00640 1 -- start start 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5f4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 0x7fe5f015fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5f4a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe5f00cbc30 0x7fe5f00cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5f4a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f00cee80 con 0x7fe5f00731e0 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5f4a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f00cf160 con 0x7fe5f00cbc30 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5f4a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f00cf440 con 0x7fe5f00b0350 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe5f00cbc30 0x7fe5f00cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe5f00cbc30 0x7fe5f00cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51802/0 (socket says 172.21.15.90:51802) 2024-03-31T15:54:22.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 -- 172.21.15.90:0/2496000732 learned_addr learned my addr 172.21.15.90:0/2496000732 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:22.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ee800640 1 --2- 172.21.15.90:0/2496000732 >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 0x7fe5f015fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 -- 172.21.15.90:0/2496000732 >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 msgr2=0x7fe5f015fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:22.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 --2- 172.21.15.90:0/2496000732 >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 0x7fe5f015fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 -- 172.21.15.90:0/2496000732 >> v1:172.21.15.102:6789/0 conn(0x7fe5f00731e0 legacy=0x7fe5f00cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:22.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ef200640 1 -- 172.21.15.90:0/2496000732 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f00cf720 con 0x7fe5f00cbc30 2024-03-31T15:54:22.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.174+0000 7fe5ee800640 1 --2- 172.21.15.90:0/2496000732 >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 0x7fe5f015fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5ef200640 1 --2- 172.21.15.90:0/2496000732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00cff20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e50 tx=0x7fe5e801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8851bd453bb47ec server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5ed400640 1 -- 172.21.15.90:0/2496000732 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80a57d0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5ed400640 1 -- 172.21.15.90:0/2496000732 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e80a5970 con 0x7fe5f00cbc30 2024-03-31T15:54:22.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5ed400640 1 -- 172.21.15.90:0/2496000732 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80a5d40 con 0x7fe5f00cbc30 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- 172.21.15.90:0/2496000732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 msgr2=0x7fe5f00cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/2496000732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00cff20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e50 tx=0x7fe5e801c8f0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- 172.21.15.90:0/2496000732 shutdown_connections 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/2496000732 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00cff20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/2496000732 >> v2:172.21.15.102:3300/0 conn(0x7fe5f00b0350 0x7fe5f015fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- 172.21.15.90:0/2496000732 >> 172.21.15.90:0/2496000732 conn(0x7fe5f00cd6b0 msgr2=0x7fe5f00cdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- 172.21.15.90:0/2496000732 shutdown_connections 2024-03-31T15:54:22.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- 172.21.15.90:0/2496000732 wait complete. 2024-03-31T15:54:22.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 Processor -- start 2024-03-31T15:54:22.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- start start 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5f00731e0 0x7fe5f0154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5f00b0350 0x7fe5f00c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00d09f0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00d0870 con 0x7fe5f00b0350 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.178+0000 7fe5f4a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f015e0c0 con 0x7fe5f00731e0 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51808/0 (socket says 172.21.15.90:51808) 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 -- 172.21.15.90:0/1601691145 learned_addr learned my addr 172.21.15.90:0/1601691145 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ee800640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5f00731e0 0x7fe5f0154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ede00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5f00b0350 0x7fe5f00c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 -- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5f00731e0 msgr2=0x7fe5f0154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5f00731e0 0x7fe5f0154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 -- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5f00b0350 msgr2=0x7fe5f00c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5f00b0350 0x7fe5f00c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 -- 172.21.15.90:0/1601691145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f00c9010 con 0x7fe5f00cbc30 2024-03-31T15:54:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5ef200640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe5e801a520 tx=0x7fe5e8019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80bc020 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e80b8670 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f00c92f0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80b8a60 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f0155470 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f00c94c0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.182+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe5e80bf020 con 0x7fe5f00cbc30 2024-03-31T15:54:22.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.186+0000 7fe5e6a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 0x7fe5d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.186+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe5e8121ff0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.186+0000 7fe5ee800640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 0x7fe5d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.186+0000 7fe5ee800640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 0x7fe5d006a840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000ab60 tx=0x7fe5e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.190+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe5e80c6030 con 0x7fe5f00cbc30 2024-03-31T15:54:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.346+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe5f0068d00 con 0x7fe5f00cbc30 2024-03-31T15:54:22.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5e6a00640 1 -- 172.21.15.90:0/1601691145 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe5e80165a0 con 0x7fe5f00cbc30 2024-03-31T15:54:22.344 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 msgr2=0x7fe5d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 0x7fe5d006a840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000ab60 tx=0x7fe5e0008040 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 msgr2=0x7fe5f00b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe5e801a520 tx=0x7fe5e8019690 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 shutdown_connections 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe5d0068380 0x7fe5d006a840 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f00cbc30 0x7fe5f00b0790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5f00b0350 0x7fe5f00c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 --2- 172.21.15.90:0/1601691145 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5f00731e0 0x7fe5f0154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 >> 172.21.15.90:0/1601691145 conn(0x7fe5f00cd6b0 msgr2=0x7fe5f0158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 shutdown_connections 2024-03-31T15:54:22.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.350+0000 7fe5f4a00640 1 -- 172.21.15.90:0/1601691145 wait complete. 2024-03-31T15:54:22.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:22.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.506+0000 7f5c34e00640 1 Processor -- start 2024-03-31T15:54:22.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.506+0000 7f5c34e00640 1 -- start start 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c34e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5c301456c0 0x7f5c3006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c34e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c3006bb30 0x7f5c300cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c34e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c300d0590 con 0x7f5c300cfde0 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c34e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5c3006bf90 con 0x7f5c301456c0 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c34e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3006e7a0 con 0x7f5c3006bb30 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5c301456c0 0x7f5c3006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5c300cfde0 0x7f5c300721f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:34120/0 (socket says 172.21.15.90:34120) 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2fe00640 1 -- 172.21.15.90:0/1368592893 learned_addr learned my addr 172.21.15.90:0/1368592893 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2ea00640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c3006bb30 0x7f5c300cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2f400640 1 -- 172.21.15.90:0/1368592893 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c3006bb30 msgr2=0x7f5c300cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2f400640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c3006bb30 0x7f5c300cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2f400640 1 -- 172.21.15.90:0/1368592893 >> v1:172.21.15.90:6789/0 conn(0x7f5c300cfde0 legacy=0x7f5c300721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2f400640 1 -- 172.21.15.90:0/1368592893 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c300d0870 con 0x7f5c301456c0 2024-03-31T15:54:22.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.510+0000 7f5c2e000640 1 -- 172.21.15.90:0/1368592893 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557342770 0 0) 0x7f5c300d0590 con 0x7f5c300cfde0 2024-03-31T15:54:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c2f400640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c301456c0 0x7f5c3006b5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c200097f0 tx=0x7f5c2001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd852cbf2e7054b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c2e000640 1 -- 172.21.15.90:0/1368592893 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c200b1070 con 0x7f5c301456c0 2024-03-31T15:54:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c2e000640 1 -- 172.21.15.90:0/1368592893 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c200a8050 con 0x7f5c301456c0 2024-03-31T15:54:22.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c2e000640 1 -- 172.21.15.90:0/1368592893 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c200ac860 con 0x7f5c301456c0 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 -- 172.21.15.90:0/1368592893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c301456c0 msgr2=0x7f5c3006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c301456c0 0x7f5c3006b5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5c200097f0 tx=0x7f5c2001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 -- 172.21.15.90:0/1368592893 shutdown_connections 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c3006bb30 0x7f5c300cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 --2- 172.21.15.90:0/1368592893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c301456c0 0x7f5c3006b5f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 -- 172.21.15.90:0/1368592893 >> 172.21.15.90:0/1368592893 conn(0x7f5c300d0c60 msgr2=0x7f5c300d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 -- 172.21.15.90:0/1368592893 shutdown_connections 2024-03-31T15:54:22.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.514+0000 7f5c34e00640 1 -- 172.21.15.90:0/1368592893 wait complete. 2024-03-31T15:54:22.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 Processor -- start 2024-03-31T15:54:22.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- start start 2024-03-31T15:54:22.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 0x7f5c3015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 0x7f5c3015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30168aa0 con 0x7f5c300cfde0 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c300d0230 con 0x7f5c3006bb30 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5c30168c20 con 0x7f5c301456c0 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51830/0 (socket says 172.21.15.90:51830) 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 -- 172.21.15.90:0/3044844345 learned_addr learned my addr 172.21.15.90:0/3044844345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2fe00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 0x7f5c3015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 -- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 msgr2=0x7f5c3015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2f400640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 0x7f5c3015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 0x7f5c3015af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 -- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 msgr2=0x7f5c3015a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 0x7f5c3015a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 -- 172.21.15.90:0/3044844345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3015e5d0 con 0x7f5c300cfde0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2fe00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 0x7f5c3015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2f400640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 0x7f5c3015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c2ea00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00d800 tx=0x7f5c1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00b6c0 con 0x7f5c300cfde0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c1c019070 con 0x7f5c300cfde0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c300c6f10 con 0x7f5c300cfde0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00ba60 con 0x7f5c300cfde0 2024-03-31T15:54:22.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c300c6150 con 0x7f5c300cfde0 2024-03-31T15:54:22.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.518+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c30068060 con 0x7f5c300cfde0 2024-03-31T15:54:22.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.522+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c1c00bc00 con 0x7f5c300cfde0 2024-03-31T15:54:22.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.522+0000 7f5c27400640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 0x7f5c1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.522+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5c1c0871a0 con 0x7f5c300cfde0 2024-03-31T15:54:22.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.522+0000 7f5c2f400640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 0x7f5c1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.522+0000 7f5c2f400640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 0x7f5c1006a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c200ae000 tx=0x7f5c20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.526+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c1c057420 con 0x7f5c300cfde0 2024-03-31T15:54:22.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.682+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5c30075180 con 0x7f5c300cfde0 2024-03-31T15:54:22.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c27400640 1 -- 172.21.15.90:0/3044844345 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f5c1c056d20 con 0x7f5c300cfde0 2024-03-31T15:54:22.680 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 msgr2=0x7f5c1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 0x7f5c1006a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c200ae000 tx=0x7f5c20008040 comp rx=0 tx=0).stop 2024-03-31T15:54:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 msgr2=0x7f5c300c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00d800 tx=0x7f5c1c00dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 shutdown_connections 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c10068380 0x7f5c1006a840 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c301456c0 0x7f5c3015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c300cfde0 0x7f5c300c7350 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 --2- 172.21.15.90:0/3044844345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c3006bb30 0x7f5c3015a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 >> 172.21.15.90:0/3044844345 conn(0x7f5c300d0c60 msgr2=0x7f5c300d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 shutdown_connections 2024-03-31T15:54:22.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.686+0000 7f5c34e00640 1 -- 172.21.15.90:0/3044844345 wait complete. 2024-03-31T15:54:22.702 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:54:22.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 Processor -- start 2024-03-31T15:54:22.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 -- start start 2024-03-31T15:54:22.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc24161500 0x7fdc24072c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc240b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc2406e9f0 con 0x7fdc240b4170 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdc24073290 con 0x7fdc24161500 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc29000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2406ec90 con 0x7fdc240cfbe0 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc24161500 0x7fdc24072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdc24161500 0x7fdc24072c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51634/0 (socket says 172.21.15.90:51634) 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 -- 172.21.15.90:0/1111493823 learned_addr learned my addr 172.21.15.90:0/1111493823 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23e00640 1 --1- 172.21.15.90:0/1111493823 >> v1:172.21.15.90:6789/0 conn(0x7fdc240b4170 0x7fdc2406b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48026/0 (socket says 172.21.15.90:48026) 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc1be00640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc240b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 -- 172.21.15.90:0/1111493823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 msgr2=0x7fdc240b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc240b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 -- 172.21.15.90:0/1111493823 >> v1:172.21.15.90:6789/0 conn(0x7fdc240b4170 legacy=0x7fdc2406b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc23400640 1 -- 172.21.15.90:0/1111493823 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2406ef70 con 0x7fdc24161500 2024-03-31T15:54:22.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.870+0000 7fdc1be00640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc240b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc23400640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc24072c00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009870 tx=0x7fdc1401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8ca26ba1c4a798 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc22a00640 1 -- 172.21.15.90:0/1111493823 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140b0070 con 0x7fdc24161500 2024-03-31T15:54:22.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc22a00640 1 -- 172.21.15.90:0/1111493823 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc140ab600 con 0x7fdc24161500 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc22a00640 1 -- 172.21.15.90:0/1111493823 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140ab9d0 con 0x7fdc24161500 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 -- 172.21.15.90:0/1111493823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 msgr2=0x7fdc24072c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc24072c00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009870 tx=0x7fdc1401c4d0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 -- 172.21.15.90:0/1111493823 shutdown_connections 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc240b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 --2- 172.21.15.90:0/1111493823 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc24072c00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 -- 172.21.15.90:0/1111493823 >> 172.21.15.90:0/1111493823 conn(0x7fdc2406c700 msgr2=0x7fdc2406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 -- 172.21.15.90:0/1111493823 shutdown_connections 2024-03-31T15:54:22.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.874+0000 7fdc29000640 1 -- 172.21.15.90:0/1111493823 wait complete. 2024-03-31T15:54:22.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 Processor -- start 2024-03-31T15:54:22.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- start start 2024-03-31T15:54:22.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc2415bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc2415d630 0x7fdc2415fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406be80 con 0x7fdc24161500 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2406bd00 con 0x7fdc2415d630 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdc2416ad20 con 0x7fdc240cfbe0 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51638/0 (socket says 172.21.15.90:51638) 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 -- 172.21.15.90:0/2102146560 learned_addr learned my addr 172.21.15.90:0/2102146560 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc23e00640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc2415d630 0x7fdc2415fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc23400640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc2415bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 -- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 msgr2=0x7fdc2415bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc2415bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 -- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc2415d630 msgr2=0x7fdc2415fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc2415d630 0x7fdc2415fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:22.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 -- 172.21.15.90:0/2102146560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc24159ef0 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1be00640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00e810 tx=0x7fdc0c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c017070 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc0c00c8d0 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00f040 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2415dac0 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.878+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2415e080 con 0x7fdc24161500 2024-03-31T15:54:22.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc24068d00 con 0x7fdc24161500 2024-03-31T15:54:22.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdc0c01a070 con 0x7fdc24161500 2024-03-31T15:54:22.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc1b400640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 0x7fdc0006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:22.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdc0c086830 con 0x7fdc24161500 2024-03-31T15:54:22.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc23400640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 0x7fdc0006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:22.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.882+0000 7fdc23400640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 0x7fdc0006a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc140ad000 tx=0x7fdc140a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:22.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:22.886+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdc0c056b40 con 0x7fdc24161500 2024-03-31T15:54:23.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc240b3da0 con 0x7fdc24161500 2024-03-31T15:54:23.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc1b400640 1 -- 172.21.15.90:0/2102146560 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fdc0c056440 con 0x7fdc24161500 2024-03-31T15:54:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:23.073 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:54:23.074 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 msgr2=0x7fdc0006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 0x7fdc0006a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc140ad000 tx=0x7fdc140a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 msgr2=0x7fdc241599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00e810 tx=0x7fdc0c00ecd0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 shutdown_connections 2024-03-31T15:54:23.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdc00068380 0x7fdc0006a840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc2415d630 0x7fdc2415fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc24161500 0x7fdc241599b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 --2- 172.21.15.90:0/2102146560 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdc240cfbe0 0x7fdc2415bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 >> 172.21.15.90:0/2102146560 conn(0x7fdc2406c700 msgr2=0x7fdc2415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 shutdown_connections 2024-03-31T15:54:23.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.078+0000 7fdc29000640 1 -- 172.21.15.90:0/2102146560 wait complete. 2024-03-31T15:54:23.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:23.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.234+0000 7f6ccd600640 1 Processor -- start 2024-03-31T15:54:23.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.234+0000 7f6ccd600640 1 -- start start 2024-03-31T15:54:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6ccd600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6ccd600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 0x7f6cc815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6ccd600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cc815f780 con 0x7f6cc806c7b0 2024-03-31T15:54:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6ccd600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6cc81622b0 con 0x7f6cc806d7c0 2024-03-31T15:54:23.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6ccd600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc815f1c0 con 0x7f6cc815e1b0 2024-03-31T15:54:23.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 0x7f6cc815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:43484/0 (socket says 172.21.15.90:43484) 2024-03-31T15:54:23.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 0x7f6cc815e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:34590/0 (socket says 172.21.15.90:34590) 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 -- 172.21.15.90:0/2571128879 learned_addr learned my addr 172.21.15.90:0/2571128879 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cbfe00640 1 --1- 172.21.15.90:0/2571128879 >> v1:172.21.15.102:6789/0 conn(0x7f6cc806c7b0 0x7f6cc806d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:48716/0 (socket says 172.21.15.90:48716) 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 -- 172.21.15.90:0/2571128879 >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 msgr2=0x7f6cc815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 --2- 172.21.15.90:0/2571128879 >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 0x7f6cc815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6000640 1 -- 172.21.15.90:0/2571128879 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779353295 0 0) 0x7f6cc815f780 con 0x7f6cc806c7b0 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 -- 172.21.15.90:0/2571128879 >> v1:172.21.15.102:6789/0 conn(0x7f6cc806c7b0 legacy=0x7f6cc806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.238+0000 7f6cc6a00640 1 -- 172.21.15.90:0/2571128879 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc815f4a0 con 0x7f6cc815e1b0 2024-03-31T15:54:23.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6cc6a00640 1 --2- 172.21.15.90:0/2571128879 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009870 tx=0x7f6cb801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6732babe4a3ae705 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6cc6000640 1 -- 172.21.15.90:0/2571128879 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80aa600 con 0x7f6cc815e1b0 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6cc6000640 1 -- 172.21.15.90:0/2571128879 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb80af070 con 0x7f6cc815e1b0 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6cc6000640 1 -- 172.21.15.90:0/2571128879 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80aa9c0 con 0x7f6cc815e1b0 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- 172.21.15.90:0/2571128879 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 msgr2=0x7f6cc8162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 --2- 172.21.15.90:0/2571128879 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009870 tx=0x7f6cb801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- 172.21.15.90:0/2571128879 shutdown_connections 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 --2- 172.21.15.90:0/2571128879 >> v2:172.21.15.102:3300/0 conn(0x7f6cc806d7c0 0x7f6cc815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 --2- 172.21.15.90:0/2571128879 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc815e1b0 0x7f6cc8162740 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- 172.21.15.90:0/2571128879 >> 172.21.15.90:0/2571128879 conn(0x7f6cc809aa40 msgr2=0x7f6cc809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:23.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- 172.21.15.90:0/2571128879 shutdown_connections 2024-03-31T15:54:23.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- 172.21.15.90:0/2571128879 wait complete. 2024-03-31T15:54:23.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 Processor -- start 2024-03-31T15:54:23.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.242+0000 7f6ccd600640 1 -- start start 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc806c7b0 0x7f6cc80afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cc815e1b0 0x7f6cc80b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc80d2940 con 0x7f6cc806d7c0 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc80d27c0 con 0x7f6cc815e1b0 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc80d2ac0 con 0x7f6cc806c7b0 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51650/0 (socket says 172.21.15.90:51650) 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 -- 172.21.15.90:0/4245375519 learned_addr learned my addr 172.21.15.90:0/4245375519 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cc6a00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc806c7b0 0x7f6cc80afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cc7400640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cc815e1b0 0x7f6cc80b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 -- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc806c7b0 msgr2=0x7f6cc80afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc806c7b0 0x7f6cc80afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 -- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cc815e1b0 msgr2=0x7f6cc80b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cc815e1b0 0x7f6cc80b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 -- 172.21.15.90:0/4245375519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc80b5080 con 0x7f6cc806d7c0 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbfe00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6cb000c480 tx=0x7f6cb000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0018070 con 0x7f6cc806d7c0 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb00074e0 con 0x7f6cc806d7c0 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc80b5320 con 0x7f6cc806d7c0 2024-03-31T15:54:23.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb00049f0 con 0x7f6cc806d7c0 2024-03-31T15:54:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc80b58e0 con 0x7f6cc806d7c0 2024-03-31T15:54:23.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.246+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc80686d0 con 0x7f6cc806d7c0 2024-03-31T15:54:23.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.250+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6cb0004b90 con 0x7f6cc806d7c0 2024-03-31T15:54:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.250+0000 7f6cbea00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 0x7f6ca406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.250+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6cb0086000 con 0x7f6cc806d7c0 2024-03-31T15:54:23.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.250+0000 7f6cc6a00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 0x7f6ca406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.250+0000 7f6cc6a00640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 0x7f6ca406a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80ae000 tx=0x7f6cb80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.254+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6cb0055c40 con 0x7f6cc806d7c0 2024-03-31T15:54:23.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.414+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cc80b1b60 con 0x7f6cc806d7c0 2024-03-31T15:54:23.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.414+0000 7f6cbea00640 1 -- 172.21.15.90:0/4245375519 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18478 (secure 0 0 0) 0x7f6cb0059f40 con 0x7f6cc806d7c0 2024-03-31T15:54:23.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:23.411 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":25,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:53:52.162960+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:23.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 msgr2=0x7f6ca406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 0x7f6ca406a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80ae000 tx=0x7f6cb80a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 msgr2=0x7f6cc80b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6cb000c480 tx=0x7f6cb000c940 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 shutdown_connections 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ca4068380 0x7f6ca406a840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cc815e1b0 0x7f6cc80b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cc806d7c0 0x7f6cc80b4b40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 --2- 172.21.15.90:0/4245375519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6cc806c7b0 0x7f6cc80afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 >> 172.21.15.90:0/4245375519 conn(0x7f6cc809aa40 msgr2=0x7f6cc80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 shutdown_connections 2024-03-31T15:54:23.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.418+0000 7f6ccd600640 1 -- 172.21.15.90:0/4245375519 wait complete. 2024-03-31T15:54:23.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:54:23.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 Processor -- start 2024-03-31T15:54:23.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 -- start start 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d415c120 0x7ff7d415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7d40d2680 0x7ff7d40b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d4160350 con 0x7ff7d40b2c20 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d4160bf0 con 0x7ff7d40d2680 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7dc600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d4160630 con 0x7ff7d415c120 2024-03-31T15:54:23.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7da600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff7d40b2c20 0x7ff7d40cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48036/0 (socket says 172.21.15.90:48036) 2024-03-31T15:54:23.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7da600640 1 -- 172.21.15.90:0/3018051785 learned_addr learned my addr 172.21.15.90:0/3018051785 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:23.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9200640 1 --2- 172.21.15.90:0/3018051785 >> v2:172.21.15.102:3300/0 conn(0x7ff7d40d2680 0x7ff7d40b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9c00640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d415c120 0x7ff7d415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d3e00640 1 -- 172.21.15.90:0/3018051785 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1564008881 0 0) 0x7ff7d4160350 con 0x7ff7d40b2c20 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d3e00640 1 -- 172.21.15.90:0/3018051785 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7bc003530 con 0x7ff7d40b2c20 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9200640 1 -- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d415c120 msgr2=0x7ff7d415fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9200640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d415c120 0x7ff7d415fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9200640 1 -- 172.21.15.90:0/3018051785 >> v1:172.21.15.90:6789/0 conn(0x7ff7d40b2c20 legacy=0x7ff7d40cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.574+0000 7ff7d9200640 1 -- 172.21.15.90:0/3018051785 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d4160910 con 0x7ff7d40d2680 2024-03-31T15:54:23.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7d9200640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d40d2680 0x7ff7d40b26e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00097c0 tx=0x7ff7c001c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f75ad78ffc94cf7c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7d3e00640 1 -- 172.21.15.90:0/3018051785 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00b0070 con 0x7ff7d40d2680 2024-03-31T15:54:23.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7d3e00640 1 -- 172.21.15.90:0/3018051785 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c0004230 con 0x7ff7d40d2680 2024-03-31T15:54:23.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7d3e00640 1 -- 172.21.15.90:0/3018051785 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0003620 con 0x7ff7d40d2680 2024-03-31T15:54:23.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 -- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d40d2680 msgr2=0x7ff7d40b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d40d2680 0x7ff7d40b26e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00097c0 tx=0x7ff7c001c350 comp rx=0 tx=0).stop 2024-03-31T15:54:23.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 -- 172.21.15.90:0/3018051785 shutdown_connections 2024-03-31T15:54:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d40d2680 0x7ff7d40b26e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 --2- 172.21.15.90:0/3018051785 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d415c120 0x7ff7d415fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 -- 172.21.15.90:0/3018051785 >> 172.21.15.90:0/3018051785 conn(0x7ff7d409aa40 msgr2=0x7ff7d409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 -- 172.21.15.90:0/3018051785 shutdown_connections 2024-03-31T15:54:23.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.578+0000 7ff7dc600640 1 -- 172.21.15.90:0/3018051785 wait complete. 2024-03-31T15:54:23.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 Processor -- start 2024-03-31T15:54:23.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- start start 2024-03-31T15:54:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 0x7ff7d4153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 0x7ff7d406f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41691c0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4169040 con 0x7ff7d415c120 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d4169340 con 0x7ff7d40d2680 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51654/0 (socket says 172.21.15.90:51654) 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 -- 172.21.15.90:0/203069506 learned_addr learned my addr 172.21.15.90:0/203069506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7da600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 0x7ff7d406f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9200640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 0x7ff7d4153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 -- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 msgr2=0x7ff7d4153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 0x7ff7d4153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 -- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 msgr2=0x7ff7d406f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 0x7ff7d406f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 -- 172.21.15.90:0/203069506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d4154820 con 0x7ff7d40b2c20 2024-03-31T15:54:23.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7da600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 0x7ff7d406f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9200640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 0x7ff7d4153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d9c00640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800e870 tx=0x7ff7c800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8017070 con 0x7ff7d40b2c20 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c800c940 con 0x7ff7d40b2c20 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d40b4fd0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800ccf0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.582+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d40b5590 con 0x7ff7d40b2c20 2024-03-31T15:54:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d40686d0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff7c801a070 con 0x7ff7d40b2c20 2024-03-31T15:54:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7d1600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 0x7ff7b406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff7c8086bf0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7d9200640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 0x7ff7b406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:23.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.586+0000 7ff7d9200640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 0x7ff7b406a870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00ad000 tx=0x7ff7c00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:23.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.590+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff7c8056de0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.898+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7d40751c0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7d1600640 1 -- 172.21.15.90:0/203069506 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7ff7c80566e0 con 0x7ff7d40b2c20 2024-03-31T15:54:23.896 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:23.896 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:54:23.896 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 msgr2=0x7ff7b406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 0x7ff7b406a870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00ad000 tx=0x7ff7c00a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 msgr2=0x7ff7d40cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800e870 tx=0x7ff7c800ec70 comp rx=0 tx=0).stop 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 shutdown_connections 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7b40683b0 0x7ff7b406a870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7d415c120 0x7ff7d406f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7d40d2680 0x7ff7d4153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 --2- 172.21.15.90:0/203069506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7d40b2c20 0x7ff7d40cb8f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:23.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 >> 172.21.15.90:0/203069506 conn(0x7ff7d409aa40 msgr2=0x7ff7d4074a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:23.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 shutdown_connections 2024-03-31T15:54:23.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:23.902+0000 7ff7dc600640 1 -- 172.21.15.90:0/203069506 wait complete. 2024-03-31T15:54:24.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:53:47.115176+0000', 'modified': '2024-03-31T15:54:11.211162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6835/3620557044', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 3620557044}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 3620557044}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-03-31T15:54:24.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:54:24.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:54:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 Processor -- start 2024-03-31T15:54:24.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 -- start start 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f70c80ce910 0x7f70c80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 0x7f70c80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c80cf230 con 0x7f70c80cfe90 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f70c80cd460 con 0x7f70c80ce910 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cf400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f70c80cd740 con 0x7f70c815d240 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f70c80ce910 0x7f70c80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f70c80ce910 0x7f70c80cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51666/0 (socket says 172.21.15.90:51666) 2024-03-31T15:54:24.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.462+0000 7f70cca00640 1 -- 172.21.15.90:0/1727673433 learned_addr learned my addr 172.21.15.90:0/1727673433 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70c7e00640 1 --2- 172.21.15.90:0/1727673433 >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 0x7f70c80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70cca00640 1 -- 172.21.15.90:0/1727673433 >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 msgr2=0x7f70c80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70cca00640 1 --2- 172.21.15.90:0/1727673433 >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 0x7f70c80ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70cca00640 1 -- 172.21.15.90:0/1727673433 >> v1:172.21.15.102:6789/0 conn(0x7f70c80cfe90 legacy=0x7f70c80d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70cca00640 1 -- 172.21.15.90:0/1727673433 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c80cda20 con 0x7f70c80ce910 2024-03-31T15:54:24.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70c7e00640 1 --2- 172.21.15.90:0/1727673433 >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 0x7f70c80ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:24.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70cca00640 1 --2- 172.21.15.90:0/1727673433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80ce910 0x7f70c80cecf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70c8072f50 tx=0x7f70bc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc6fcbf7adbbe0f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70c7400640 1 -- 172.21.15.90:0/1727673433 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0b0070 con 0x7f70c80ce910 2024-03-31T15:54:24.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70c7400640 1 -- 172.21.15.90:0/1727673433 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70bc0ab600 con 0x7f70c80ce910 2024-03-31T15:54:24.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.466+0000 7f70c7400640 1 -- 172.21.15.90:0/1727673433 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0ab9d0 con 0x7f70c80ce910 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- 172.21.15.90:0/1727673433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80ce910 msgr2=0x7f70c80cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- 172.21.15.90:0/1727673433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80ce910 0x7f70c80cecf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f70c8072f50 tx=0x7f70bc01c420 comp rx=0 tx=0).stop 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- 172.21.15.90:0/1727673433 shutdown_connections 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- 172.21.15.90:0/1727673433 >> v2:172.21.15.102:3300/0 conn(0x7f70c815d240 0x7f70c80ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- 172.21.15.90:0/1727673433 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80ce910 0x7f70c80cecf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- 172.21.15.90:0/1727673433 >> 172.21.15.90:0/1727673433 conn(0x7f70c808bfd0 msgr2=0x7f70c80d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- 172.21.15.90:0/1727673433 shutdown_connections 2024-03-31T15:54:24.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- 172.21.15.90:0/1727673433 wait complete. 2024-03-31T15:54:24.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 Processor -- start 2024-03-31T15:54:24.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- start start 2024-03-31T15:54:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70c80ce910 0x7f70c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70c815d240 0x7f70c814a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70c80cd1d0 con 0x7f70c80cfe90 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70c80cd050 con 0x7f70c80ce910 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cf400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f70c8169410 con 0x7f70c815d240 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51670/0 (socket says 172.21.15.90:51670) 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 -- 172.21.15.90:0/3127020128 learned_addr learned my addr 172.21.15.90:0/3127020128 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cca00640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70c80ce910 0x7f70c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70cd400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70c815d240 0x7f70c814a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 -- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70c815d240 msgr2=0x7f70c814a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70c815d240 0x7f70c814a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 -- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70c80ce910 msgr2=0x7f70c806e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.470+0000 7f70c7e00640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70c80ce910 0x7f70c806e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c7e00640 1 -- 172.21.15.90:0/3127020128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c8153c90 con 0x7f70c80cfe90 2024-03-31T15:54:24.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c7e00640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c820 tx=0x7f70b400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c4c00640 1 -- 172.21.15.90:0/3127020128 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4015070 con 0x7f70c80cfe90 2024-03-31T15:54:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c4c00640 1 -- 172.21.15.90:0/3127020128 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b40079d0 con 0x7f70c80cfe90 2024-03-31T15:54:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c8153f70 con 0x7f70c80cfe90 2024-03-31T15:54:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c4c00640 1 -- 172.21.15.90:0/3127020128 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4007de0 con 0x7f70c80cfe90 2024-03-31T15:54:24.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c8154530 con 0x7f70c80cfe90 2024-03-31T15:54:24.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.474+0000 7f70c4c00640 1 -- 172.21.15.90:0/3127020128 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f70b4018070 con 0x7f70c80cfe90 2024-03-31T15:54:24.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 Processor -- start 2024-03-31T15:54:24.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- start start 2024-03-31T15:54:24.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c4c00640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 0x7f70a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c4c00640 1 -- 172.21.15.90:0/3127020128 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f70b4086a80 con 0x7f70c80cfe90 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cd400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 0x7f70a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927962880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927916490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557927768950 con 0x557927962dc0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557927705990 con 0x5579279624a0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x557927764900 con 0x5579279160b0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51676/0 (socket says 172.21.15.90:51676) 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 -- 172.21.15.90:0/2689575022 learned_addr learned my addr 172.21.15.90:0/2689575022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cd400640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927916490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cca00640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927962880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cd400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 0x7f70a806a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70c814bb90 tx=0x7f70c0009430 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 -- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 msgr2=0x557927916490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927916490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 -- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 msgr2=0x557927962880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927962880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 -- 172.21.15.90:0/2689575022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579277ad100 con 0x557927962dc0 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cd400640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927916490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70cca00640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927962880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f70b4084ca0 tx=0x7f70b40568c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f2bbde0450b590 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70b9600640 1 -- 172.21.15.90:0/2689575022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b401f040 con 0x557927962dc0 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70b9600640 1 -- 172.21.15.90:0/2689575022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400ce10 con 0x557927962dc0 2024-03-31T15:54:24.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70b9600640 1 -- 172.21.15.90:0/2689575022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4052db0 con 0x557927962dc0 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- 172.21.15.90:0/2689575022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 msgr2=0x557927915b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f70b4084ca0 tx=0x7f70b40568c0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- 172.21.15.90:0/2689575022 shutdown_connections 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927916490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927915b70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 --2- 172.21.15.90:0/2689575022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927962880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- 172.21.15.90:0/2689575022 >> 172.21.15.90:0/2689575022 conn(0x557927800b80 msgr2=0x557927945460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- 172.21.15.90:0/2689575022 shutdown_connections 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.478+0000 7f70d291c000 1 -- 172.21.15.90:0/2689575022 wait complete. 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- start start 2024-03-31T15:54:24.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 Processor -- start 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279160b0 0x5579279310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279624a0 0x557927931630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x557927767c60 con 0x557927962dc0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5579277263e0 con 0x5579279160b0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x557927775060 con 0x5579279624a0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 10 client.0 _ll_register_callbacks cb 0x5579277c90f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51678/0 (socket says 172.21.15.90:51678) 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 -- 172.21.15.90:0/1191451714 learned_addr learned my addr 172.21.15.90:0/1191451714 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cca00640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279160b0 0x5579279310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70c7e00640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279624a0 0x557927931630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 -- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279624a0 msgr2=0x557927931630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279624a0 0x557927931630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 -- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279160b0 msgr2=0x5579279310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279160b0 0x5579279310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 -- 172.21.15.90:0/1191451714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579277328c0 con 0x557927962dc0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cd400640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70c0019e70 tx=0x7f70c0004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f89398356194c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0027060 con 0x557927962dc0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f70c0012040 con 0x557927962dc0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b400ce10 con 0x557927962dc0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f70c0022410 con 0x557927962dc0 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 10 client.4824 fetch_fsmap learned FSMap version 9 2024-03-31T15:54:24.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5579276f0e80 con 0x557927962dc0 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400ce10 con 0x557927962dc0 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f70c0030080 con 0x557927962dc0 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 10 client.4824 fetch_fsmap finished waiting for FSMap version 9 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 10 client.4824 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 20 client.4824 populate_metadata read hostname 'smithi090' 2024-03-31T15:54:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x557927932410 0x557927974a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 4 client.4824 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 --> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557927703bc0 con 0x557927932410 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cca00640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x557927932410 0x557927974a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70cca00640 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x557927932410 0x557927974a20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70f1db9d694cd020 server_cookie=459e433b9c282fb5 in_seq=0 out_seq=0 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.482+0000 7f70ae000640 10 client.4824 ms_handle_connect on v2:172.21.15.90:6834/3620557044 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70ae000640 1 -- 172.21.15.90:0/1191451714 <== mds.0 v2:172.21.15.90:6834/3620557044 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557927703bc0 con 0x557927932410 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70ae000640 10 client.4824 handle_command_reply: tid=0 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 client.4824 shutdown 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 msgr2=0x557927931b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70c0019e70 tx=0x7f70c0004a80 comp rx=0 tx=0).stop 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 shutdown_connections 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x557927932410 0x557927974a20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927931b70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279624a0 0x557927931630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 --2- 172.21.15.90:0/1191451714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279160b0 0x5579279310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70ae000640 0 client.4824 ms_handle_reset on v2:172.21.15.90:6834/3620557044 2024-03-31T15:54:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 >> 172.21.15.90:0/1191451714 conn(0x557927800b80 msgr2=0x557927915270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 shutdown_connections 2024-03-31T15:54:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 1 -- 172.21.15.90:0/1191451714 wait complete. 2024-03-31T15:54:24.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.486+0000 7f70d291c000 20 client.4824 trim_cache size 0 max 16384 2024-03-31T15:54:24.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 Processor -- start 2024-03-31T15:54:24.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 -- start start 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927960760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 0x5579279611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8204f70 con 0x5579279624a0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8204020 con 0x557927962dc0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f70c8155fc0 con 0x5579279160b0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51688/0 (socket says 172.21.15.90:51688) 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.502+0000 7f70c7e00640 1 -- 172.21.15.90:0/2034863875 learned_addr learned my addr 172.21.15.90:0/2034863875 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 0x5579279611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cca00640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927960760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 -- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 msgr2=0x557927960760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927960760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 -- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 msgr2=0x5579279611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 0x5579279611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 -- 172.21.15.90:0/2034863875 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557927723490 con 0x5579279624a0 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 0x5579279611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cca00640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927960760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f70b4084d50 tx=0x7f70b401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86255eda66594e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70b8c00640 1 -- 172.21.15.90:0/2034863875 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4072070 con 0x5579279624a0 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70b8c00640 1 -- 172.21.15.90:0/2034863875 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400ce10 con 0x5579279624a0 2024-03-31T15:54:24.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70b8c00640 1 -- 172.21.15.90:0/2034863875 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b401f040 con 0x5579279624a0 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- 172.21.15.90:0/2034863875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 msgr2=0x557927960ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f70b4084d50 tx=0x7f70b401c000 comp rx=0 tx=0).stop 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- 172.21.15.90:0/2034863875 shutdown_connections 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557927962dc0 0x5579279611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5579279624a0 0x557927960ca0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- 172.21.15.90:0/2034863875 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x557927960760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- 172.21.15.90:0/2034863875 >> 172.21.15.90:0/2034863875 conn(0x557927800b80 msgr2=0x55792796f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- 172.21.15.90:0/2034863875 shutdown_connections 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- 172.21.15.90:0/2034863875 wait complete. 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- start start 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 Processor -- start 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x5579279d2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927974a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55792776cf80 con 0x557927962dc0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70c82039d0 con 0x5579279624a0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x55792770cc40 con 0x5579279160b0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70d291c000 10 client.0 _ll_register_callbacks cb 0x7f70c0009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51702/0 (socket says 172.21.15.90:51702) 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 -- 172.21.15.90:0/2821652450 learned_addr learned my addr 172.21.15.90:0/2821652450 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70c7e00640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927974a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cca00640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x5579279d2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 -- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 msgr2=0x5579279d2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x5579279d2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 -- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 msgr2=0x557927974a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:24.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927974a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 -- 172.21.15.90:0/2821652450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557927721e80 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70cd400640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f70c000a010 tx=0x7f70c0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda9558c79bd7671 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.506+0000 7f70ae000640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0003b90 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f70c0003d30 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70b400ce10 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f70c003e050 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 10 client.4830 fetch_fsmap learned FSMap version 9 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x55792770f270 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b400ce10 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f70c0041020 con 0x557927962dc0 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 10 client.4830 fetch_fsmap finished waiting for FSMap version 9 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 10 client.4830 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 20 client.4830 populate_metadata read hostname 'smithi090' 2024-03-31T15:54:24.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x5579279754a0 0x557927975880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 4 client.4830 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 --> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] -- command(tid 0: {"prefix": "status"}) -- 0x557927775e90 con 0x5579279754a0 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cca00640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x5579279754a0 0x557927975880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cca00640 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x5579279754a0 0x557927975880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=114a21a46ea53a70 server_cookie=d64bb9d1b1fd9902 in_seq=0 out_seq=0 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 10 client.4830 ms_handle_connect on v2:172.21.15.90:6834/3620557044 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 1 -- 172.21.15.90:0/2821652450 <== mds.0 v2:172.21.15.90:6834/3620557044 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x557927775e90 con 0x5579279754a0 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 10 client.4830 handle_command_reply: tid=0 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 client.4830 shutdown 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 msgr2=0x557927974f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f70c000a010 tx=0x7f70c0011000 comp rx=0 tx=0).stop 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 shutdown_connections 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:6834/3620557044,v1:172.21.15.90:6835/3620557044] conn(0x5579279754a0 0x557927975880 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x557927962dc0 0x557927974f60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5579279624a0 0x557927974a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 --2- 172.21.15.90:0/2821652450 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x5579279160b0 0x5579279d2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70ae000640 0 client.4830 ms_handle_reset on v2:172.21.15.90:6834/3620557044 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 >> 172.21.15.90:0/2821652450 conn(0x557927800b80 msgr2=0x55792796fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 shutdown_connections 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 1 -- 172.21.15.90:0/2821652450 wait complete. 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70d291c000 20 client.4830 trim_cache size 0 max 16384 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "0135ca30-606a-448f-b61f-3ca22bf9710a", 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4293, 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-31T15:54:24.510 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 36.765040646000003, 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 9, 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 25, 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 25, 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 41.443639855000001 2024-03-31T15:54:24.511 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 msgr2=0x7f70a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 0x7f70a806a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f70c814bb90 tx=0x7f70c0009430 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 msgr2=0x7f70c8159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c820 tx=0x7f70b400cce0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 shutdown_connections 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f70a8068380 0x7f70a806a840 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70c815d240 0x7f70c814a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70c80cfe90 0x7f70c8159290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 --2- 172.21.15.90:0/3127020128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70c80ce910 0x7f70c806e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:24.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 >> 172.21.15.90:0/3127020128 conn(0x7f70c808bfd0 msgr2=0x7f70c8070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:24.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 shutdown_connections 2024-03-31T15:54:24.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:24.510+0000 7f70cf400640 1 -- 172.21.15.90:0/3127020128 wait complete. 2024-03-31T15:54:24.526 INFO:teuthology.run_tasks:Running task sequential... 2024-03-31T15:54:24.535 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-31T15:54:24.535 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-31T15:54:24.535 INFO:teuthology.task.print:Enabling mgr modules 2024-03-31T15:54:24.536 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-31T15:54:24.545 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-31T15:54:24.545 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-31T15:54:24.545 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:24.546 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:24.546 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:24.546 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-31T15:54:24.546 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:24.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:24.547 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-31T15:54:24.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:24.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-31T15:54:24.624 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-03-31T15:54:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:54:24.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:24.627 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:54:24.636 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-31T15:54:24.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:24.637 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-31T15:54:24.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:24.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-03-31T15:54:24.898 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-31T15:54:24.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:54:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.038+0000 7f367dc00640 1 Processor -- start 2024-03-31T15:54:25.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.038+0000 7f367dc00640 1 -- start start 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f367dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f367806bf30 0x7f367815ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f367dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36780b47e0 con 0x7f367815bfd0 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f367dc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36780b4ac0 con 0x7f36781449a0 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f367dc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f367815b7d0 con 0x7f367806bf30 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f366fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f367815bfd0 0x7f367806b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48038/0 (socket says 172.21.15.90:48038) 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f366fe00640 1 -- 172.21.15.90:0/1670251 learned_addr learned my addr 172.21.15.90:0/1670251 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3677400640 1 --2- 172.21.15.90:0/1670251 >> v2:172.21.15.102:3300/0 conn(0x7f367806bf30 0x7f367815ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016261995 0 0) 0x7f36780b47e0 con 0x7f367815bfd0 2024-03-31T15:54:25.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f365c003530 con 0x7f367815bfd0 2024-03-31T15:54:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3677400640 1 -- 172.21.15.90:0/1670251 >> v1:172.21.15.102:6789/0 conn(0x7f36781449a0 legacy=0x7f36780b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3677400640 1 -- 172.21.15.90:0/1670251 >> v1:172.21.15.90:6789/0 conn(0x7f367815bfd0 legacy=0x7f367806b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2410965756 0 0) 0x7f365c003530 con 0x7f367815bfd0 2024-03-31T15:54:25.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.042+0000 7f3677400640 1 -- 172.21.15.90:0/1670251 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367815bab0 con 0x7f367806bf30 2024-03-31T15:54:25.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f3677400640 1 --2- 172.21.15.90:0/1670251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f367815ea50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3670009d90 tx=0x7f367001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dacd084eaa8844c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700b2070 con 0x7f367806bf30 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36700a55e0 con 0x7f367806bf30 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f3676000640 1 -- 172.21.15.90:0/1670251 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660001680 con 0x7f367806bf30 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 -- 172.21.15.90:0/1670251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 msgr2=0x7f367815ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 --2- 172.21.15.90:0/1670251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f367815ea50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3670009d90 tx=0x7f367001c890 comp rx=0 tx=0).stop 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 -- 172.21.15.90:0/1670251 shutdown_connections 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 --2- 172.21.15.90:0/1670251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f367815ea50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 -- 172.21.15.90:0/1670251 >> 172.21.15.90:0/1670251 conn(0x7f36780b1c80 msgr2=0x7f36780b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:25.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 -- 172.21.15.90:0/1670251 shutdown_connections 2024-03-31T15:54:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.046+0000 7f367dc00640 1 -- 172.21.15.90:0/1670251 wait complete. 2024-03-31T15:54:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 Processor -- start 2024-03-31T15:54:25.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- start start 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f36780aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f367815bfd0 0x7f3678139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36780b2e00 con 0x7f36781449a0 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36780b2c80 con 0x7f367806bf30 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f36780b2f80 con 0x7f367815bfd0 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:25.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f3676a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f36780aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51712/0 (socket says 172.21.15.90:51712) 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 -- 172.21.15.90:0/4223084722 learned_addr learned my addr 172.21.15.90:0/4223084722 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f3677400640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f367815bfd0 0x7f3678139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 -- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f367815bfd0 msgr2=0x7f3678139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f367815bfd0 0x7f3678139450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 -- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 msgr2=0x7f36780aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f36780aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 -- 172.21.15.90:0/4223084722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36781f7330 con 0x7f36781449a0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366fe00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f366000c520 tx=0x7f366000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3660017070 con 0x7f36781449a0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3660005d10 con 0x7f36781449a0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36600072b0 con 0x7f36781449a0 2024-03-31T15:54:25.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36781fae20 con 0x7f36781449a0 2024-03-31T15:54:25.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36781fb3e0 con 0x7f36781449a0 2024-03-31T15:54:25.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.050+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36780686d0 con 0x7f36781449a0 2024-03-31T15:54:25.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.054+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f366001a070 con 0x7f36781449a0 2024-03-31T15:54:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.054+0000 7f366ea00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 0x7f365406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.054+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3660086100 con 0x7f36781449a0 2024-03-31T15:54:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.054+0000 7f3676a00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 0x7f365406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:25.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.054+0000 7f3676a00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 0x7f365406a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36780aea80 tx=0x7f3668009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:25.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.058+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3660055d20 con 0x7f36781449a0 2024-03-31T15:54:25.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.250+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3678166bd0 con 0x7f36781449a0 2024-03-31T15:54:25.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.250+0000 7f366ea00640 1 -- 172.21.15.90:0/4223084722 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f366005c030 con 0x7f36781449a0 2024-03-31T15:54:25.246 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.250+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 msgr2=0x7f365406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.250+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 0x7f365406a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36780aea80 tx=0x7f3668009040 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 msgr2=0x7f3678140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f366000c520 tx=0x7f366000c9e0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 shutdown_connections 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3654068380 0x7f365406a840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f367815bfd0 0x7f3678139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36781449a0 0x7f3678140370 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 --2- 172.21.15.90:0/4223084722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f367806bf30 0x7f36780aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 >> 172.21.15.90:0/4223084722 conn(0x7f36780b1c80 msgr2=0x7f367806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:25.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 shutdown_connections 2024-03-31T15:54:25.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25.254+0000 7f367dc00640 1 -- 172.21.15.90:0/4223084722 wait complete. 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:54:25.267 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:54:25.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:25.268 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:54:25.277 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:54:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42458sec preferred_lft 42458sec 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:54:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:25.279 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T15:54:25.279 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T15:54:25.280 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:25.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:25.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:25.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:25.430 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:54:25.448 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:54:25.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:25.450 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:25.507 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:25.508 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:25.508 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:54:25.508 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:25.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:25.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:25.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:25.645 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:54:25.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:25.701 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-31T15:54:25.752 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:25.752 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:25.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:25.752 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-31T15:54:25.752 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:25.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:25.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:25.878 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-31T15:54:25.878 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-31T15:54:25.879 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:25.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:26.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:26.021 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:26.021 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:26.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:54:26.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T15:54:26.022 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:26.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:26.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:26.146 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:54:26.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:26.146 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:26.155 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:54:26.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:26.155 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:26.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:54:26.266 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:54:26.316 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:26.317 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:26.318 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:26.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:26.319 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:54:26.370 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:54:26.371 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:54:26.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:54:26.435 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:26.488 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:54:26.488 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:54:26.488 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:54:26.489 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:26.490 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:26.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:26.491 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:54:26.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:54:26.514+0000 7f6870c7e580 -1 init, newargv = 0x55f88ef8b210 newargc=15 2024-03-31T15:54:26.509 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[35287]: starting ceph client 2024-03-31T15:54:26.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[35287]: starting fuse 2024-03-31T15:54:26.518 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:54:26.519 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:54:26.519 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:54:26.520 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:54:26.639 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.35287.asok 2024-03-31T15:54:26.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:26.650 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.35287.asok 2024-03-31T15:54:26.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:26.651 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.35287.asok status 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "35287", 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:54:26.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4842, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4842 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 410409902 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:26.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 410409902 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4842 192.168.0.1:0/410409902", 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/410409902", 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 9, 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 25, 2024-03-31T15:54:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:54:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:54:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:54:26.826 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:54:26.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:54:26.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 Processor -- start 2024-03-31T15:54:26.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 -- start start 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc7d415b010 0x7fc7d415f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 0x7fc7d4072820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d415eb10 con 0x7fc7d40723c0 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d415b3f0 con 0x7fc7d415b010 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7dbc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d40708b0 con 0x7fc7d406d2a0 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc7d415b010 0x7fc7d415f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc7d40723c0 0x7fc7d406cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48052/0 (socket says 172.21.15.90:48052) 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc7d415b010 0x7fc7d415f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51718/0 (socket says 172.21.15.90:51718) 2024-03-31T15:54:26.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d3e00640 1 -- 172.21.15.90:0/3518446310 learned_addr learned my addr 172.21.15.90:0/3518446310 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9c00640 1 --2- 172.21.15.90:0/3518446310 >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 0x7fc7d4072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 -- 172.21.15.90:0/3518446310 >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 msgr2=0x7fc7d4072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 --2- 172.21.15.90:0/3518446310 >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 0x7fc7d4072820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 -- 172.21.15.90:0/3518446310 >> v1:172.21.15.90:6789/0 conn(0x7fc7d40723c0 legacy=0x7fc7d406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9200640 1 -- 172.21.15.90:0/3518446310 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d4072f30 con 0x7fc7d415b010 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d3400640 1 -- 172.21.15.90:0/3518446310 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793462595 0 0) 0x7fc7d415eb10 con 0x7fc7d40723c0 2024-03-31T15:54:26.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.978+0000 7fc7d9c00640 1 --2- 172.21.15.90:0/3518446310 >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 0x7fc7d4072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7d9200640 1 --2- 172.21.15.90:0/3518446310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d415b010 0x7fc7d415f5b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40b6ef0 tx=0x7fc7c801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d33f540878752f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7d3400640 1 -- 172.21.15.90:0/3518446310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80b0070 con 0x7fc7d415b010 2024-03-31T15:54:26.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7d3400640 1 -- 172.21.15.90:0/3518446310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c80ab600 con 0x7fc7d415b010 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7d3400640 1 -- 172.21.15.90:0/3518446310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80ab9d0 con 0x7fc7d415b010 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 -- 172.21.15.90:0/3518446310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d415b010 msgr2=0x7fc7d415f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/3518446310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d415b010 0x7fc7d415f5b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40b6ef0 tx=0x7fc7c801c480 comp rx=0 tx=0).stop 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 -- 172.21.15.90:0/3518446310 shutdown_connections 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/3518446310 >> v2:172.21.15.102:3300/0 conn(0x7fc7d406d2a0 0x7fc7d4072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/3518446310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d415b010 0x7fc7d415f5b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:26.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 -- 172.21.15.90:0/3518446310 >> 172.21.15.90:0/3518446310 conn(0x7fc7d40d13a0 msgr2=0x7fc7d40d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 -- 172.21.15.90:0/3518446310 shutdown_connections 2024-03-31T15:54:26.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.982+0000 7fc7dbc00640 1 -- 172.21.15.90:0/3518446310 wait complete. 2024-03-31T15:54:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 Processor -- start 2024-03-31T15:54:26.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- start start 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d40723c0 0x7fc7d41592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc7d415b010 0x7fc7d4159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d40d05d0 con 0x7fc7d406d2a0 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d40d0450 con 0x7fc7d40723c0 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d40d0750 con 0x7fc7d415b010 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51732/0 (socket says 172.21.15.90:51732) 2024-03-31T15:54:26.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 -- 172.21.15.90:0/1224564300 learned_addr learned my addr 172.21.15.90:0/1224564300 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 -- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc7d415b010 msgr2=0x7fc7d4159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc7d415b010 0x7fc7d4159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 -- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d40723c0 msgr2=0x7fc7d41592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d40723c0 0x7fc7d41592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 -- 172.21.15.90:0/1224564300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d40c7c30 con 0x7fc7d406d2a0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d9200640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80ae040 tx=0x7fc7c8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80b0040 con 0x7fc7d406d2a0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c80ba070 con 0x7fc7d406d2a0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80b4830 con 0x7fc7d406d2a0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d415b680 con 0x7fc7d406d2a0 2024-03-31T15:54:26.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.986+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d415bc40 con 0x7fc7d406d2a0 2024-03-31T15:54:26.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.990+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d40686d0 con 0x7fc7d406d2a0 2024-03-31T15:54:26.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.990+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc7c80ab4c0 con 0x7fc7d406d2a0 2024-03-31T15:54:26.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.990+0000 7fc7d0c00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 0x7fc7b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:26.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.990+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc7c8120c30 con 0x7fc7d406d2a0 2024-03-31T15:54:26.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.990+0000 7fc7d3e00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 0x7fc7b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:26.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.994+0000 7fc7d3e00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 0x7fc7b406a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0007fd0 tx=0x7fc7c0007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:26.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:26.994+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc7c8016070 con 0x7fc7d406d2a0 2024-03-31T15:54:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.186+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc7d40cf6d0 con 0x7fc7d406d2a0 2024-03-31T15:54:27.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.186+0000 7fc7d0c00640 1 -- 172.21.15.90:0/1224564300 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fc7c80f0d80 con 0x7fc7d406d2a0 2024-03-31T15:54:27.182 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 msgr2=0x7fc7b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 0x7fc7b406a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0007fd0 tx=0x7fc7c0007960 comp rx=0 tx=0).stop 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 msgr2=0x7fc7d406e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80ae040 tx=0x7fc7c8019220 comp rx=0 tx=0).stop 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 shutdown_connections 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc7b4068380 0x7fc7b406a840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc7d415b010 0x7fc7d4159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:27.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d40723c0 0x7fc7d41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:27.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 --2- 172.21.15.90:0/1224564300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc7d406d2a0 0x7fc7d406e6d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:27.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 >> 172.21.15.90:0/1224564300 conn(0x7fc7d40d13a0 msgr2=0x7fc7d40b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:27.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 shutdown_connections 2024-03-31T15:54:27.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27.190+0000 7fc7dbc00640 1 -- 172.21.15.90:0/1224564300 wait complete. 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:27.203 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:54:27.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:54:27.204 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:54:27.204 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-31T15:54:27.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.204 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:54:27.213 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:27.213 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:27.213 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:27.213 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:54:27.214 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42456sec preferred_lft 42456sec 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:54:27.215 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:27.216 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-03-31T15:54:27.216 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.216 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:54:27.272 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:54:27.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.274 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:27.327 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:54:27.328 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:27.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:27.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:27.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.467 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:54:27.475 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:54:27.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.477 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-03-31T15:54:27.531 INFO:teuthology.orchestra.run.smithi090.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-31T15:54:27.532 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:27.532 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:27.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:54:27.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-31T15:54:27.532 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:27.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:27.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:27.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-31T15:54:27.677 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:27.677 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:27.677 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 92:58:74:fb:c4:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9058:74ff:fefb:c436/64 scope link tentative 2024-03-31T15:54:27.678 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:27.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 92:58:74:fb:c4:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:54:27.748 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:54:27.749 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.749 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::9058:74ff:fefb:c436/64 scope link tentative 2024-03-31T15:54:27.749 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:27.750 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-31T15:54:27.750 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:27.750 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:27.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-31T15:54:27.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-31T15:54:27.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-31T15:54:27.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-31T15:54:27.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-31T15:54:27.751 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:27.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:27.939 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:27.939 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:54:27.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.1 up 2024-03-31T15:54:27.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.1 master ceph-brx 2024-03-31T15:54:27.939 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:54:28.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:28.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:28.065 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-31T15:54:28.065 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:28.066 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:28.074 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-31T15:54:28.074 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:28.074 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:28.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:54:28.183 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:28.232 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:54:28.232 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:54:28.232 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:54:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:54:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:54:28.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:54:28.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:28.236 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:54:28.287 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:54:28.287 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-31T15:54:28.287 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-31T15:54:28.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:54:28.350 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:54:28.402 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:54:28.403 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:54:28.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:54:28.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:28.405 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:54:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:2024-03-31T15:54:28.430+0000 7f808724d580 -1 init, newargv = 0x56228d3560d0 newargc=15 2024-03-31T15:54:28.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:ceph-fuse[35426]: starting ceph client 2024-03-31T15:54:28.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi090.stderr:ceph-fuse[35426]: starting fuse 2024-03-31T15:54:28.434 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:54:28.434 INFO:teuthology.orchestra.run.smithi090.stdout:44 2024-03-31T15:54:28.434 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:54:28.435 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.2")) 2024-03-31T15:54:28.436 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:54:28.556 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.2.35426.asok 2024-03-31T15:54:28.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:28.567 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.35426.asok 2024-03-31T15:54:28.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:28.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.35426.asok status 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "2", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "35426", 2024-03-31T15:54:28.738 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 4860, 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 4860 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2844999534 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:54:28.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2844999534 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.4860 192.168.0.2:0/2844999534", 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.2:0/2844999534", 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 9, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 25, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:54:28.740 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:54:28.753 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:54:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 Processor -- start 2024-03-31T15:54:28.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 -- start start 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a7006c7b0 0x7f6a7006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a7006d5f0 0x7f6a7015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a7015f780 con 0x7f6a7015e1b0 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6a701622b0 con 0x7f6a7006d5f0 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a77a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6a7006da20 con 0x7f6a7006c7b0 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a75a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a7006d5f0 0x7f6a7015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a75a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6a7006d5f0 0x7f6a7015e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51744/0 (socket says 172.21.15.90:51744) 2024-03-31T15:54:28.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a75a00640 1 -- 172.21.15.90:0/4033726910 learned_addr learned my addr 172.21.15.90:0/4033726910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:28.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.894+0000 7f6a6fe00640 1 --2- 172.21.15.90:0/4033726910 >> v2:172.21.15.102:3300/0 conn(0x7f6a7006c7b0 0x7f6a7006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a75a00640 1 -- 172.21.15.90:0/4033726910 >> v2:172.21.15.102:3300/0 conn(0x7f6a7006c7b0 msgr2=0x7f6a7006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:28.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a75a00640 1 --2- 172.21.15.90:0/4033726910 >> v2:172.21.15.102:3300/0 conn(0x7f6a7006c7b0 0x7f6a7006d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:28.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a75a00640 1 -- 172.21.15.90:0/4033726910 >> v1:172.21.15.102:6789/0 conn(0x7f6a7015e1b0 legacy=0x7f6a70162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:28.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a75a00640 1 -- 172.21.15.90:0/4033726910 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7015f1c0 con 0x7f6a7006d5f0 2024-03-31T15:54:28.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a75a00640 1 --2- 172.21.15.90:0/4033726910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006d5f0 0x7f6a7015e590 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a7006ed50 tx=0x7f6a6801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4967707a26764328 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:28.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a6f400640 1 -- 172.21.15.90:0/4033726910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a680a57f0 con 0x7f6a7006d5f0 2024-03-31T15:54:28.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a6f400640 1 -- 172.21.15.90:0/4033726910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a680a5990 con 0x7f6a7006d5f0 2024-03-31T15:54:28.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a6f400640 1 -- 172.21.15.90:0/4033726910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a680a5d60 con 0x7f6a7006d5f0 2024-03-31T15:54:28.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a77a00640 1 -- 172.21.15.90:0/4033726910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006d5f0 msgr2=0x7f6a7015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.898+0000 7f6a77a00640 1 --2- 172.21.15.90:0/4033726910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006d5f0 0x7f6a7015e590 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a7006ed50 tx=0x7f6a6801c930 comp rx=0 tx=0).stop 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- 172.21.15.90:0/4033726910 shutdown_connections 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 --2- 172.21.15.90:0/4033726910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006d5f0 0x7f6a7015e590 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 --2- 172.21.15.90:0/4033726910 >> v2:172.21.15.102:3300/0 conn(0x7f6a7006c7b0 0x7f6a7006d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- 172.21.15.90:0/4033726910 >> 172.21.15.90:0/4033726910 conn(0x7f6a7009aa40 msgr2=0x7f6a7009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- 172.21.15.90:0/4033726910 shutdown_connections 2024-03-31T15:54:28.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- 172.21.15.90:0/4033726910 wait complete. 2024-03-31T15:54:28.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 Processor -- start 2024-03-31T15:54:28.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- start start 2024-03-31T15:54:28.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a7006d5f0 0x7f6a7015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a7015e1b0 0x7f6a700b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700d2940 con 0x7f6a7006c7b0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6a700d27c0 con 0x7f6a7015e1b0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a77a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6a700d2ac0 con 0x7f6a7006d5f0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51750/0 (socket says 172.21.15.90:51750) 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 -- 172.21.15.90:0/355767730 learned_addr learned my addr 172.21.15.90:0/355767730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a7015e1b0 0x7f6a700b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a6fe00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a7006d5f0 0x7f6a7015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 -- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a7006d5f0 msgr2=0x7f6a7015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a7006d5f0 0x7f6a7015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 -- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a7015e1b0 msgr2=0x7f6a700b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:28.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a7015e1b0 0x7f6a700b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 -- 172.21.15.90:0/355767730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a70155c40 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a75000640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c370 tx=0x7f6a6400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.902+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a640052d0 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a64005470 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64007260 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a70158890 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a70158e50 con 0x7f6a7006c7b0 2024-03-31T15:54:28.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a700686d0 con 0x7f6a7006c7b0 2024-03-31T15:54:28.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a64007400 con 0x7f6a7006c7b0 2024-03-31T15:54:28.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6cc00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 0x7f6a5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:28.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6a64086290 con 0x7f6a7006c7b0 2024-03-31T15:54:28.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.906+0000 7f6a6fe00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 0x7f6a5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:28.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.910+0000 7f6a6fe00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 0x7f6a5006a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6a700c8a10 tx=0x7f6a5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:28.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:28.910+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6a640563e0 con 0x7f6a7006c7b0 2024-03-31T15:54:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6a70169f90 con 0x7f6a7006c7b0 2024-03-31T15:54:29.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a6cc00640 1 -- 172.21.15.90:0/355767730 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f6a64055ce0 con 0x7f6a7006c7b0 2024-03-31T15:54:29.124 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T15:54:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 msgr2=0x7f6a5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 0x7f6a5006a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6a700c8a10 tx=0x7f6a5c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 msgr2=0x7f6a70155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:29.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c370 tx=0x7f6a6400c830 comp rx=0 tx=0).stop 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 shutdown_connections 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a500683b0 0x7f6a5006a870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a7015e1b0 0x7f6a700b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a7006d5f0 0x7f6a7015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 --2- 172.21.15.90:0/355767730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a7006c7b0 0x7f6a70155700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 >> 172.21.15.90:0/355767730 conn(0x7f6a7009aa40 msgr2=0x7f6a70162740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 shutdown_connections 2024-03-31T15:54:29.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:29.130+0000 7f6a77a00640 1 -- 172.21.15.90:0/355767730 wait complete. 2024-03-31T15:54:29.145 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:54:29.145 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:54:29.145 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-31T15:54:29.145 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-31T15:54:29.145 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:29.146 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:54:29.146 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:54:29.146 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:54:29.146 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-31T15:54:29.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:29.146 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:29.156 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42425sec preferred_lft 42425sec 2024-03-31T15:54:29.157 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-31T15:54:29.158 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:29.158 INFO:teuthology.orchestra.run.smithi102.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:54:29.158 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 02:42:70:92:89:06 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:29.158 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:54:29.158 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:29.159 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-31T15:54:29.159 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:29.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:29.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:29.326 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:54:29.345 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-31T15:54:29.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:29.347 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-31T15:54:29.419 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-31T15:54:29.419 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:29.420 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:54:29.420 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:29.420 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:29.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:54:29.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:54:29.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:54:29.421 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:29.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:29.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:29.554 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-31T15:54:29.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:29.563 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-31T15:54:29.613 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:29.613 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:29.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:29.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-31T15:54:29.613 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:29.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:29.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:29.739 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-31T15:54:29.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-31T15:54:29.740 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:29.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:29.894 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:29.894 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:29.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-31T15:54:29.894 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-03-31T15:54:29.894 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:29.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:30.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:30.021 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-31T15:54:30.021 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:30.022 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:30.030 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-31T15:54:30.031 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:30.031 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:30.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-31T15:54:30.140 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:30.193 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-31T15:54:30.193 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-31T15:54:30.194 INFO:teuthology.orchestra.run.smithi102.stdout:/run/lock 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/fs/nfsd 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/run/rpc_pipefs 2024-03-31T15:54:30.195 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:30.196 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:30.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:30.197 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-31T15:54:30.253 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:54:30.253 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-31T15:54:30.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-31T15:54:30.321 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/run/lock 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-31T15:54:30.378 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/fs/nfsd 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/run/rpc_pipefs 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-31T15:54:30.379 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:30.380 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:30.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:30.381 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-31T15:54:30.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi102.stderr:2024-03-31T15:54:30.397+0000 7fc8ffdbb580 -1 init, newargv = 0x5588b7384400 newargc=15 2024-03-31T15:54:30.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi102.stderr:ceph-fuse[33492]: starting ceph client 2024-03-31T15:54:30.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi102.stderr:ceph-fuse[33492]: starting fuse 2024-03-31T15:54:31.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-31T15:54:31.524 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:31.578 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-31T15:54:31.578 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-31T15:54:31.578 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-31T15:54:31.578 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-31T15:54:31.578 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/run/lock 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/fs/nfsd 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:31.579 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/rpc_pipefs 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:31.580 INFO:teuthology.orchestra.run.smithi102.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:31.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:31.581 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-31T15:54:31.632 INFO:teuthology.orchestra.run.smithi102.stdout:43 2024-03-31T15:54:31.633 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> import glob 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> import re 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> import os 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> import subprocess 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> def _find_admin_socket(client_name): 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> files = glob.glob(asok_path) 2024-03-31T15:54:31.633 DEBUG:teuthology.orchestra.run.smithi102:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> # Given a non-glob path, it better be there 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> if "*" not in asok_path: 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> assert(len(files) == 1) 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> return files[0] 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> for f in files: 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> contents = proc_f.read() 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> if mountpoint in contents: 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> return f 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:54:31.634 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:31.635 DEBUG:teuthology.orchestra.run.smithi102:> print(_find_admin_socket("client.1")) 2024-03-31T15:54:31.635 DEBUG:teuthology.orchestra.run.smithi102:> ' 2024-03-31T15:54:31.762 INFO:teuthology.orchestra.run.smithi102.stdout:/var/run/ceph/ceph-client.1.33492.asok 2024-03-31T15:54:31.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:31.773 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33492.asok 2024-03-31T15:54:31.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:31.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33492.asok status 2024-03-31T15:54:31.944 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-31T15:54:31.944 INFO:teuthology.orchestra.run.smithi102.stdout: "metadata": { 2024-03-31T15:54:31.944 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:54:31.944 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "entity_id": "1", 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": "smithi102", 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "pid": "33492", 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "root": "/" 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_count": 0, 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_pinned_count": 0, 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4801, 2024-03-31T15:54:31.945 INFO:teuthology.orchestra.run.smithi102.stdout: "inst": { 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "name": { 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "client", 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4801 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 2695542190 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:54:31.946 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 2695542190 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "inst_str": "client.4801 192.168.0.1:0/2695542190", 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "addr_str": "192.168.0.1:0/2695542190", 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "inode_count": 1, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_epoch": 9, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch": 26, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "blocklisted": false, 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs" 2024-03-31T15:54:31.947 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-31T15:54:31.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:54:32.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 Processor -- start 2024-03-31T15:54:32.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 -- start start 2024-03-31T15:54:32.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffb6c145a10 0x7ffb6c071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c15fb80 0x7ffb6c0cc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffb6c160000 0x7ffb6c0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6c0ce7f0 con 0x7ffb6c145a10 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6c145df0 con 0x7ffb6c160000 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb71000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6c1605d0 con 0x7ffb6c15fb80 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb6b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffb6c145a10 0x7ffb6c071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb6b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffb6c145a10 0x7ffb6c071590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51754/0 (socket says 172.21.15.90:51754) 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb6b400640 1 -- 172.21.15.90:0/2430399467 learned_addr learned my addr 172.21.15.90:0/2430399467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb63e00640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c15fb80 0x7ffb6c0cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.102+0000 7ffb6be00640 1 --2- 172.21.15.90:0/2430399467 >> v2:172.21.15.102:3300/0 conn(0x7ffb6c160000 0x7ffb6c0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 -- 172.21.15.90:0/2430399467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c15fb80 msgr2=0x7ffb6c0cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c15fb80 0x7ffb6c0cc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 -- 172.21.15.90:0/2430399467 >> v2:172.21.15.102:3300/0 conn(0x7ffb6c160000 msgr2=0x7ffb6c0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:32.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2430399467 >> v2:172.21.15.102:3300/0 conn(0x7ffb6c160000 0x7ffb6c0ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 -- 172.21.15.90:0/2430399467 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c06ce30 con 0x7ffb6c145a10 2024-03-31T15:54:32.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c145a10 0x7ffb6c071590 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0683b0 tx=0x7ffb5c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dda6baf665f229 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6aa00640 1 -- 172.21.15.90:0/2430399467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0b0070 con 0x7ffb6c145a10 2024-03-31T15:54:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6aa00640 1 -- 172.21.15.90:0/2430399467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c0ab600 con 0x7ffb6c145a10 2024-03-31T15:54:32.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb6aa00640 1 -- 172.21.15.90:0/2430399467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0ab9d0 con 0x7ffb6c145a10 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 -- 172.21.15.90:0/2430399467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c145a10 msgr2=0x7ffb6c071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c145a10 0x7ffb6c071590 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0683b0 tx=0x7ffb5c01c520 comp rx=0 tx=0).stop 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 -- 172.21.15.90:0/2430399467 shutdown_connections 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 --2- 172.21.15.90:0/2430399467 >> v2:172.21.15.102:3300/0 conn(0x7ffb6c160000 0x7ffb6c0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c15fb80 0x7ffb6c0cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 --2- 172.21.15.90:0/2430399467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c145a10 0x7ffb6c071590 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.106+0000 7ffb71000640 1 -- 172.21.15.90:0/2430399467 >> 172.21.15.90:0/2430399467 conn(0x7ffb6c0cff30 msgr2=0x7ffb6c0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- 172.21.15.90:0/2430399467 shutdown_connections 2024-03-31T15:54:32.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- 172.21.15.90:0/2430399467 wait complete. 2024-03-31T15:54:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 Processor -- start 2024-03-31T15:54:32.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- start start 2024-03-31T15:54:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb6c145a10 0x7ffb6c140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c160000 0x7ffb6c1651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c0d0db0 con 0x7ffb6c15fb80 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c0d0c30 con 0x7ffb6c145a10 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6c071ad0 con 0x7ffb6c160000 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51758/0 (socket says 172.21.15.90:51758) 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 -- 172.21.15.90:0/2176503641 learned_addr learned my addr 172.21.15.90:0/2176503641 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb6c145a10 0x7ffb6c140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb6be00640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c160000 0x7ffb6c1651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 -- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c160000 msgr2=0x7ffb6c1651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c160000 0x7ffb6c1651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 -- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb6c145a10 msgr2=0x7ffb6c140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb6c145a10 0x7ffb6c140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 -- 172.21.15.90:0/2176503641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c13e520 con 0x7ffb6c15fb80 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63e00640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400d800 tx=0x7ffb5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:32.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54018020 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb540096a0 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54009a50 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.110+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c0b3280 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c0b3840 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb6c068d00 con 0x7ffb6c15fb80 2024-03-31T15:54:32.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffb5401b020 con 0x7ffb6c15fb80 2024-03-31T15:54:32.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb63400640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 0x7ffb4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:32.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffb540872a0 con 0x7ffb6c15fb80 2024-03-31T15:54:32.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.114+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 0x7ffb4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:32.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.118+0000 7ffb6b400640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 0x7ffb4806a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0ad000 tx=0x7ffb5c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:32.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.118+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffb54022020 con 0x7ffb6c15fb80 2024-03-31T15:54:32.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.326+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ffb6c0686c0 con 0x7ffb6c15fb80 2024-03-31T15:54:32.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb63400640 1 -- 172.21.15.90:0/2176503641 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7ffb54057450 con 0x7ffb6c15fb80 2024-03-31T15:54:32.324 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-31T15:54:32.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 msgr2=0x7ffb4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:32.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 0x7ffb4806a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c0ad000 tx=0x7ffb5c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:32.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 msgr2=0x7ffb6c141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400d800 tx=0x7ffb5400dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 shutdown_connections 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb48068380 0x7ffb4806a840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb6c160000 0x7ffb6c1651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6c15fb80 0x7ffb6c141770 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 --2- 172.21.15.90:0/2176503641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb6c145a10 0x7ffb6c140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 >> 172.21.15.90:0/2176503641 conn(0x7ffb6c0cff30 msgr2=0x7ffb6c144990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 shutdown_connections 2024-03-31T15:54:32.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:32.330+0000 7ffb71000640 1 -- 172.21.15.90:0/2176503641 wait complete. 2024-03-31T15:54:32.346 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:54:32.346 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:54:32.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:54:32.347 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-31T15:54:32.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.347 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-31T15:54:32.356 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42422sec preferred_lft 42422sec 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:54:32.357 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 02:42:70:92:89:06 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:54:32.358 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.359 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:54:32.418 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-31T15:54:32.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.420 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-31T15:54:32.471 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:54:32.472 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:54:32.473 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:32.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:32.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:32.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.605 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-31T15:54:32.614 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-31T15:54:32.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.614 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-31T15:54:32.664 INFO:teuthology.orchestra.run.smithi102.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-31T15:54:32.665 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:32.665 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:32.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:54:32.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-31T15:54:32.665 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:32.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:32.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:32.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-31T15:54:32.811 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:32.811 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether c6:ad:fb:84:4a:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::c4ad:fbff:fe84:4a9e/64 scope link 2024-03-31T15:54:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:32.813 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether c6:ad:fb:84:4a:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-31T15:54:32.876 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::c4ad:fbff:fe84:4a9e/64 scope link 2024-03-31T15:54:32.877 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:54:32.878 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-31T15:54:32.878 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:32.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:33.061 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:54:33.061 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-31T15:54:33.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.1 up 2024-03-31T15:54:33.061 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.1 master ceph-brx 2024-03-31T15:54:33.061 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-31T15:54:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:54:33.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:33.143 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-31T15:54:33.143 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:33.143 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:33.196 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-31T15:54:33.197 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:54:33.197 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:33.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-31T15:54:33.308 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:33.361 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-31T15:54:33.361 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/run/lock 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-31T15:54:33.362 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/fs/nfsd 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/run/rpc_pipefs 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:54:33.363 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:54:33.364 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:54:33.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.365 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-31T15:54:33.416 INFO:teuthology.orchestra.run.smithi102.stdout:43 2024-03-31T15:54:33.416 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-31T15:54:33.416 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-31T15:54:33.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-31T15:54:33.478 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-31T15:54:33.530 INFO:teuthology.orchestra.run.smithi102.stdout:/run/lock 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-31T15:54:33.531 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/fs/nfsd 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/run/rpc_pipefs 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-4 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-5 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-6 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-7 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-31T15:54:33.532 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:33.533 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:54:33.533 INFO:teuthology.orchestra.run.smithi102.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:33.533 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:54:33.533 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:54:33.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.534 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-31T15:54:33.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi102.stderr:2024-03-31T15:54:33.557+0000 7fa692253580 -1 init, newargv = 0x558d5d523530 newargc=15 2024-03-31T15:54:33.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi102.stderr:ceph-fuse[33611]: starting ceph client 2024-03-31T15:54:33.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi102.stderr:ceph-fuse[33611]: starting fuse 2024-03-31T15:54:33.559 INFO:teuthology.orchestra.run.smithi102.stdout:43 2024-03-31T15:54:33.559 INFO:teuthology.orchestra.run.smithi102.stdout:44 2024-03-31T15:54:33.559 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-31T15:54:33.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> import glob 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> import re 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> import os 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> import subprocess 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> def _find_admin_socket(client_name): 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> files = glob.glob(asok_path) 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> # Given a non-glob path, it better be there 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> if "*" not in asok_path: 2024-03-31T15:54:33.560 DEBUG:teuthology.orchestra.run.smithi102:> assert(len(files) == 1) 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> return files[0] 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> for f in files: 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> contents = proc_f.read() 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> if mountpoint in contents: 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> return f 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> print(_find_admin_socket("client.3")) 2024-03-31T15:54:33.561 DEBUG:teuthology.orchestra.run.smithi102:> ' 2024-03-31T15:54:33.679 INFO:teuthology.orchestra.run.smithi102.stdout:/var/run/ceph/ceph-client.3.33611.asok 2024-03-31T15:54:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-31T15:54:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:54:33.691 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.33611.asok 2024-03-31T15:54:33.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.33611.asok status 2024-03-31T15:54:33.858 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-31T15:54:33.858 INFO:teuthology.orchestra.run.smithi102.stdout: "metadata": { 2024-03-31T15:54:33.858 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:54:33.858 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:54:33.858 INFO:teuthology.orchestra.run.smithi102.stdout: "entity_id": "3", 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": "smithi102", 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "pid": "33611", 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "root": "/" 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_count": 0, 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_pinned_count": 0, 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4625, 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "inst": { 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "name": { 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "client", 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4625 2024-03-31T15:54:33.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1321502498 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.2:0", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1321502498 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "inst_str": "client.4625 192.168.0.2:0/1321502498", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "addr_str": "192.168.0.2:0/1321502498", 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "inode_count": 1, 2024-03-31T15:54:33.860 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_epoch": 9, 2024-03-31T15:54:33.861 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch": 27, 2024-03-31T15:54:33.861 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:54:33.861 INFO:teuthology.orchestra.run.smithi102.stdout: "blocklisted": false, 2024-03-31T15:54:33.861 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs" 2024-03-31T15:54:33.861 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-31T15:54:33.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.873 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:54:33.873 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:33.882 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:54:33.882 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:33.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:33.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.942 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:54:33.942 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:33.992 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:54:33.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:33.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:33.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:34.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:34.062 DEBUG:teuthology.orchestra.run.smithi102:> stat --file-system '--printf=%T 2024-03-31T15:54:34.062 DEBUG:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:34.071 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2024-03-31T15:54:34.071 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:34.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:34.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:34.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:34.135 DEBUG:teuthology.orchestra.run.smithi102:> stat --file-system '--printf=%T 2024-03-31T15:54:34.135 DEBUG:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:34.188 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2024-03-31T15:54:34.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:34.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:34.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-31T15:54:34.255 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-31T15:54:34.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 Processor -- start 2024-03-31T15:54:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 -- start start 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcb640b2a80 0x7fcb64160470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb640b2e90 con 0x7fcb640d0680 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6406de50 con 0x7fcb641457e0 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.450+0000 7fcb63200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6406d5b0 con 0x7fcb640b2a80 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb62800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcb640b2a80 0x7fcb64160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb5be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcb640d0680 0x7fcb640ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38182/0 (socket says 172.21.15.90:38182) 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb5be00640 1 -- 172.21.15.90:0/2028999231 learned_addr learned my addr 172.21.15.90:0/2028999231 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:34.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb62800640 1 -- 172.21.15.90:0/2028999231 >> v1:172.21.15.102:6789/0 conn(0x7fcb641457e0 legacy=0x7fcb640cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb62800640 1 -- 172.21.15.90:0/2028999231 >> v1:172.21.15.90:6789/0 conn(0x7fcb640d0680 legacy=0x7fcb640ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb61400640 1 -- 172.21.15.90:0/2028999231 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086789499 0 0) 0x7fcb4c001120 con 0x7fcb640d0680 2024-03-31T15:54:34.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.454+0000 7fcb62800640 1 -- 172.21.15.90:0/2028999231 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6406d890 con 0x7fcb640b2a80 2024-03-31T15:54:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb62800640 1 --2- 172.21.15.90:0/2028999231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb64160470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009d90 tx=0x7fcb5c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbd3cd924d74281 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb61400640 1 -- 172.21.15.90:0/2028999231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0b2070 con 0x7fcb640b2a80 2024-03-31T15:54:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb61400640 1 -- 172.21.15.90:0/2028999231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0a5920 con 0x7fcb640b2a80 2024-03-31T15:54:34.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb61400640 1 -- 172.21.15.90:0/2028999231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0a5cf0 con 0x7fcb640b2a80 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 -- 172.21.15.90:0/2028999231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 msgr2=0x7fcb64160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 --2- 172.21.15.90:0/2028999231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb64160470 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009d90 tx=0x7fcb5c01c970 comp rx=0 tx=0).stop 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 -- 172.21.15.90:0/2028999231 shutdown_connections 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 --2- 172.21.15.90:0/2028999231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb64160470 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 -- 172.21.15.90:0/2028999231 >> 172.21.15.90:0/2028999231 conn(0x7fcb6406b9f0 msgr2=0x7fcb6406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 -- 172.21.15.90:0/2028999231 shutdown_connections 2024-03-31T15:54:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.458+0000 7fcb63200640 1 -- 172.21.15.90:0/2028999231 wait complete. 2024-03-31T15:54:34.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 Processor -- start 2024-03-31T15:54:34.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- start start 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 0x7fcb6413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 0x7fcb6413c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcb640cf400 con 0x7fcb640b2a80 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcb640cf280 con 0x7fcb640d0680 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6416a520 con 0x7fcb641457e0 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43034/0 (socket says 172.21.15.90:43034) 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 -- 172.21.15.90:0/3334063835 learned_addr learned my addr 172.21.15.90:0/3334063835 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb5be00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 0x7fcb6413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb62800640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 0x7fcb6413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 -- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 msgr2=0x7fcb6413c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 0x7fcb6413c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 -- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 msgr2=0x7fcb6413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 0x7fcb6413c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 -- 172.21.15.90:0/3334063835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6413bad0 con 0x7fcb640b2a80 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb5be00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 0x7fcb6413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb62800640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 0x7fcb6413c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb61e00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c3c0 tx=0x7fcb5400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54017070 con 0x7fcb640b2a80 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb540052d0 con 0x7fcb640b2a80 2024-03-31T15:54:34.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb64156300 con 0x7fcb640b2a80 2024-03-31T15:54:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54003840 con 0x7fcb640b2a80 2024-03-31T15:54:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb641568c0 con 0x7fcb640b2a80 2024-03-31T15:54:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.462+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb640685f0 con 0x7fcb640b2a80 2024-03-31T15:54:34.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.466+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcb5401a070 con 0x7fcb640b2a80 2024-03-31T15:54:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.466+0000 7fcb5a000640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 0x7fcb4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.466+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcb54086110 con 0x7fcb640b2a80 2024-03-31T15:54:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.466+0000 7fcb5be00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 0x7fcb4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.466+0000 7fcb5be00640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 0x7fcb4006a840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0013c0 tx=0x7fcb4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.470+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcb54005480 con 0x7fcb640b2a80 2024-03-31T15:54:34.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb6406e9d0 con 0x7fcb640b2a80 2024-03-31T15:54:34.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb5a000640 1 -- 172.21.15.90:0/3334063835 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7fcb540562c0 con 0x7fcb640b2a80 2024-03-31T15:54:34.630 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:34.630 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 msgr2=0x7fcb4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 0x7fcb4006a840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0013c0 tx=0x7fcb4c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 msgr2=0x7fcb6413b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400c3c0 tx=0x7fcb5400c880 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 shutdown_connections 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcb40068380 0x7fcb4006a840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcb641457e0 0x7fcb6413c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb640d0680 0x7fcb6413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 --2- 172.21.15.90:0/3334063835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcb640b2a80 0x7fcb6413b590 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.634+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 >> 172.21.15.90:0/3334063835 conn(0x7fcb6406b9f0 msgr2=0x7fcb640cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:34.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.638+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 shutdown_connections 2024-03-31T15:54:34.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.638+0000 7fcb63200640 1 -- 172.21.15.90:0/3334063835 wait complete. 2024-03-31T15:54:34.652 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:34.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 Processor -- start 2024-03-31T15:54:34.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 -- start start 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43f015c290 0x7f43f00ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43f00ad3d0 0x7f43f00aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f006e9c0 con 0x7f43f00af040 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f43f006de40 con 0x7f43f015c290 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.798+0000 7f43f6400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f43f006e120 con 0x7f43f00ad3d0 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43f015c290 0x7f43f00ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43f015c290 0x7f43f00ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43040/0 (socket says 172.21.15.90:43040) 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 -- 172.21.15.90:0/2460612543 learned_addr learned my addr 172.21.15.90:0/2460612543 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43e7e00640 1 --2- 172.21.15.90:0/2460612543 >> v2:172.21.15.102:3300/0 conn(0x7f43f00ad3d0 0x7f43f00aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 -- 172.21.15.90:0/2460612543 >> v2:172.21.15.102:3300/0 conn(0x7f43f00ad3d0 msgr2=0x7f43f00aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:34.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 --2- 172.21.15.90:0/2460612543 >> v2:172.21.15.102:3300/0 conn(0x7f43f00ad3d0 0x7f43f00aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 -- 172.21.15.90:0/2460612543 >> v1:172.21.15.102:6789/0 conn(0x7f43f00af040 legacy=0x7f43f015fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 -- 172.21.15.90:0/2460612543 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f006e400 con 0x7f43f015c290 2024-03-31T15:54:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43ef400640 1 --2- 172.21.15.90:0/2460612543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f00ace90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f43f0070e00 tx=0x7f43e001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b6e31b47162868 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43eea00640 1 -- 172.21.15.90:0/2460612543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e00b0070 con 0x7f43f015c290 2024-03-31T15:54:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43eea00640 1 -- 172.21.15.90:0/2460612543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e00ab600 con 0x7f43f015c290 2024-03-31T15:54:34.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.802+0000 7f43eea00640 1 -- 172.21.15.90:0/2460612543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e00ab9d0 con 0x7f43f015c290 2024-03-31T15:54:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- 172.21.15.90:0/2460612543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 msgr2=0x7f43f00ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- 172.21.15.90:0/2460612543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f00ace90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f43f0070e00 tx=0x7f43e001c520 comp rx=0 tx=0).stop 2024-03-31T15:54:34.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- 172.21.15.90:0/2460612543 shutdown_connections 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- 172.21.15.90:0/2460612543 >> v2:172.21.15.102:3300/0 conn(0x7f43f00ad3d0 0x7f43f00aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- 172.21.15.90:0/2460612543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f00ace90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- 172.21.15.90:0/2460612543 >> 172.21.15.90:0/2460612543 conn(0x7f43f00b18b0 msgr2=0x7f43f00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- 172.21.15.90:0/2460612543 shutdown_connections 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- 172.21.15.90:0/2460612543 wait complete. 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 Processor -- start 2024-03-31T15:54:34.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- start start 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 0x7f43f0152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43f00af040 0x7f43f009c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0169000 con 0x7f43f015c290 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43f006dc00 con 0x7f43f00ad3d0 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43f6400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43f0169180 con 0x7f43f00af040 2024-03-31T15:54:34.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43046/0 (socket says 172.21.15.90:43046) 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 -- 172.21.15.90:0/33077339 learned_addr learned my addr 172.21.15.90:0/33077339 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43ef400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 0x7f43f0152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43e7e00640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43f00af040 0x7f43f009c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 -- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43f00af040 msgr2=0x7f43f009c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43f00af040 0x7f43f009c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 -- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 msgr2=0x7f43f0152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 0x7f43f0152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43efe00640 1 -- 172.21.15.90:0/33077339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f009d200 con 0x7f43f015c290 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.806+0000 7f43ef400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 0x7f43f0152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43efe00640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7a0 tx=0x7f43e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8003b80 con 0x7f43f015c290 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e8003d20 con 0x7f43f015c290 2024-03-31T15:54:34.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e800aa90 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f009d4e0 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f0152b70 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f00685f0 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f43e800ac30 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 0x7f43cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f43e8086ad0 con 0x7f43f015c290 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.810+0000 7f43ef400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 0x7f43cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:34.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.814+0000 7f43ef400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 0x7f43cc06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43e00ad000 tx=0x7f43e00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:34.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:34.814+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f43e80566d0 con 0x7f43f015c290 2024-03-31T15:54:35.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f43f006ebf0 con 0x7f43f015c290 2024-03-31T15:54:35.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43e7400640 1 -- 172.21.15.90:0/33077339 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f43e805a9d0 con 0x7f43f015c290 2024-03-31T15:54:35.033 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 msgr2=0x7f43cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 0x7f43cc06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43e00ad000 tx=0x7f43e00a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 msgr2=0x7f43f009cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c7a0 tx=0x7f43e800cc60 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 shutdown_connections 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43cc068380 0x7f43cc06a840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43f015c290 0x7f43f009cbf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43f00af040 0x7f43f009c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 --2- 172.21.15.90:0/33077339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f00ad3d0 0x7f43f0152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 >> 172.21.15.90:0/33077339 conn(0x7f43f00b18b0 msgr2=0x7f43f015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 shutdown_connections 2024-03-31T15:54:35.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.038+0000 7f43f6400640 1 -- 172.21.15.90:0/33077339 wait complete. 2024-03-31T15:54:35.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 Processor -- start 2024-03-31T15:54:35.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 -- start start 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f990806d020 0x7f99080ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990815e8c0 con 0x7f990806d490 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99080d0ff0 con 0x7f990815e4e0 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9910600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f99080d12d0 con 0x7f990806d020 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f990e600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f990806d490 0x7f99080cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38192/0 (socket says 172.21.15.90:38192) 2024-03-31T15:54:35.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f990e600640 1 -- 172.21.15.90:0/782019842 learned_addr learned my addr 172.21.15.90:0/782019842 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:35.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f990d200640 1 --2- 172.21.15.90:0/782019842 >> v2:172.21.15.102:3300/0 conn(0x7f990806d020 0x7f99080ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716570473 0 0) 0x7f990815e8c0 con 0x7f990806d490 2024-03-31T15:54:35.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.198+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98f0003530 con 0x7f990806d490 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1426447987 0 0) 0x7f98f0003530 con 0x7f990806d490 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 >> v2:172.21.15.102:3300/0 conn(0x7f990806d020 msgr2=0x7f99080ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 --2- 172.21.15.90:0/782019842 >> v2:172.21.15.102:3300/0 conn(0x7f990806d020 0x7f99080ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 >> v1:172.21.15.102:6789/0 conn(0x7f990815e4e0 legacy=0x7f990806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99080cc710 con 0x7f990806d490 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f9900001a00 con 0x7f990806d490 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 >> v1:172.21.15.90:6789/0 conn(0x7f990806d490 legacy=0x7f99080cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f00039f0 0x7f98f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 0x7f98f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0005220 0x7f98f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99080d12d0 con 0x7f98f0004810 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98f000c150 con 0x7f98f00039f0 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f98f000c350 con 0x7f98f0005220 2024-03-31T15:54:35.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 0x7f98f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990dc00640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f00039f0 0x7f98f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990e600640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0005220 0x7f98f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 -- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0005220 msgr2=0x7f98f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0005220 0x7f98f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 -- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f00039f0 msgr2=0x7f98f0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98f00039f0 0x7f98f0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 -- 172.21.15.90:0/782019842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f0003530 con 0x7f98f0004810 2024-03-31T15:54:35.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990e600640 1 -- 172.21.15.90:0/782019842 reap_dead start 2024-03-31T15:54:35.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.202+0000 7f990d200640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 0x7f98f0004c90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009ae0 tx=0x7f98f401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455b91dae81c4666 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f40096b0 con 0x7f98f0004810 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f40a64e0 con 0x7f98f0004810 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9907e00640 1 -- 172.21.15.90:0/782019842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f40a68d0 con 0x7f98f0004810 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 msgr2=0x7f98f0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 0x7f98f0004c90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009ae0 tx=0x7f98f401ca70 comp rx=0 tx=0).stop 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- 172.21.15.90:0/782019842 shutdown_connections 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 --2- 172.21.15.90:0/782019842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98f0004810 0x7f98f0004c90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- 172.21.15.90:0/782019842 >> 172.21.15.90:0/782019842 conn(0x7f99080d1f20 msgr2=0x7f99080d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- 172.21.15.90:0/782019842 shutdown_connections 2024-03-31T15:54:35.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- 172.21.15.90:0/782019842 wait complete. 2024-03-31T15:54:35.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 Processor -- start 2024-03-31T15:54:35.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- start start 2024-03-31T15:54:35.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 0x7f99081fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9908157e50 0x7f99081582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99080d0d90 con 0x7f9908159d60 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99080d0c10 con 0x7f9908157e50 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f9910600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f99080cefb0 con 0x7f98f0004810 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f990e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f990e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43072/0 (socket says 172.21.15.90:43072) 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f990e600640 1 -- 172.21.15.90:0/308565588 learned_addr learned my addr 172.21.15.90:0/308565588 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f990dc00640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 0x7f99081fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.206+0000 7f990d200640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9908157e50 0x7f99081582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 -- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 msgr2=0x7f99081fa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 0x7f99081fa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 -- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9908157e50 msgr2=0x7f99081582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9908157e50 0x7f99081582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 -- 172.21.15.90:0/308565588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990815ab20 con 0x7f9908159d60 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990dc00640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 0x7f99081fa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f990e600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f99080b7fa0 tx=0x7f9900007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990000abc0 con 0x7f9908159d60 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9900018070 con 0x7f9908159d60 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990815ae00 con 0x7f9908159d60 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000133f0 con 0x7f9908159d60 2024-03-31T15:54:35.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99080ca0e0 con 0x7f9908159d60 2024-03-31T15:54:35.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99080685f0 con 0x7f9908159d60 2024-03-31T15:54:35.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.210+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9900013590 con 0x7f9908159d60 2024-03-31T15:54:35.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.214+0000 7f9905600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 0x7f98e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.214+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f99000873e0 con 0x7f9908159d60 2024-03-31T15:54:35.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.214+0000 7f990dc00640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 0x7f98e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.214+0000 7f990dc00640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 0x7f98e806a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c10 tx=0x7f98fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.214+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9900057500 con 0x7f9908159d60 2024-03-31T15:54:35.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.374+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f990815e8c0 con 0x7f9908159d60 2024-03-31T15:54:35.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.374+0000 7f9905600640 1 -- 172.21.15.90:0/308565588 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f9900056e00 con 0x7f9908159d60 2024-03-31T15:54:35.370 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 msgr2=0x7f98e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 0x7f98e806a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c10 tx=0x7f98fc009040 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 msgr2=0x7f990815a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f99080b7fa0 tx=0x7f9900007b20 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 shutdown_connections 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e80683b0 0x7f98e806a870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9908159d60 0x7f990815a1e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9908157e50 0x7f99081582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 --2- 172.21.15.90:0/308565588 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98f0004810 0x7f99081fa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 >> 172.21.15.90:0/308565588 conn(0x7f99080d1f20 msgr2=0x7f9908070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 shutdown_connections 2024-03-31T15:54:35.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.378+0000 7f9910600640 1 -- 172.21.15.90:0/308565588 wait complete. 2024-03-31T15:54:35.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:35.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 Processor -- start 2024-03-31T15:54:35.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 -- start start 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d7415e4e0 0x7f4d7406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4d740cdf20 0x7f4d7406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7415e8c0 con 0x7f4d7406ce50 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4d740d0ff0 con 0x7f4d740cdf20 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d79800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4d740d12d0 con 0x7f4d7415e4e0 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d73400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4d740cdf20 0x7f4d7406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d72000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4d7406ce50 0x7f4d740ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38204/0 (socket says 172.21.15.90:38204) 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d72000640 1 -- 172.21.15.90:0/3763414704 learned_addr learned my addr 172.21.15.90:0/3763414704 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.534+0000 7f4d72a00640 1 --2- 172.21.15.90:0/3763414704 >> v2:172.21.15.102:3300/0 conn(0x7f4d7415e4e0 0x7f4d7406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.538+0000 7f4d73400640 1 -- 172.21.15.90:0/3763414704 >> v2:172.21.15.102:3300/0 conn(0x7f4d7415e4e0 msgr2=0x7f4d7406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.538+0000 7f4d73400640 1 --2- 172.21.15.90:0/3763414704 >> v2:172.21.15.102:3300/0 conn(0x7f4d7415e4e0 0x7f4d7406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.538+0000 7f4d73400640 1 -- 172.21.15.90:0/3763414704 >> v1:172.21.15.90:6789/0 conn(0x7f4d7406ce50 legacy=0x7f4d740ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.538+0000 7f4d73400640 1 -- 172.21.15.90:0/3763414704 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d740cc710 con 0x7f4d740cdf20 2024-03-31T15:54:35.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.538+0000 7f4d71600640 1 -- 172.21.15.90:0/3763414704 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210616075 0 0) 0x7f4d7415e8c0 con 0x7f4d7406ce50 2024-03-31T15:54:35.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d73400640 1 --2- 172.21.15.90:0/3763414704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d740cdf20 0x7f4d7406d2c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009d90 tx=0x7f4d6c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b90c5166cb5d6b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d71600640 1 -- 172.21.15.90:0/3763414704 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0b2070 con 0x7f4d740cdf20 2024-03-31T15:54:35.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d71600640 1 -- 172.21.15.90:0/3763414704 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6c0a5870 con 0x7f4d740cdf20 2024-03-31T15:54:35.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d71600640 1 -- 172.21.15.90:0/3763414704 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0a5c40 con 0x7f4d740cdf20 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- 172.21.15.90:0/3763414704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d740cdf20 msgr2=0x7f4d7406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 --2- 172.21.15.90:0/3763414704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d740cdf20 0x7f4d7406d2c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009d90 tx=0x7f4d6c01c8a0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- 172.21.15.90:0/3763414704 shutdown_connections 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 --2- 172.21.15.90:0/3763414704 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d740cdf20 0x7f4d7406d2c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 --2- 172.21.15.90:0/3763414704 >> v2:172.21.15.102:3300/0 conn(0x7f4d7415e4e0 0x7f4d7406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- 172.21.15.90:0/3763414704 >> 172.21.15.90:0/3763414704 conn(0x7f4d740d1f20 msgr2=0x7f4d740d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- 172.21.15.90:0/3763414704 shutdown_connections 2024-03-31T15:54:35.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- 172.21.15.90:0/3763414704 wait complete. 2024-03-31T15:54:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 Processor -- start 2024-03-31T15:54:35.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.542+0000 7f4d79800640 1 -- start start 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4d7406ce50 0x7f4d740ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 0x7f4d741430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7416b1b0 con 0x7f4d7415e4e0 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7416b030 con 0x7f4d740cdf20 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7416b330 con 0x7f4d7406ce50 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43084/0 (socket says 172.21.15.90:43084) 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 -- 172.21.15.90:0/2632765437 learned_addr learned my addr 172.21.15.90:0/2632765437 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d72a00640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4d7406ce50 0x7f4d740ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d72000640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 0x7f4d741430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 -- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4d7406ce50 msgr2=0x7f4d740ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4d7406ce50 0x7f4d740ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 -- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 msgr2=0x7f4d741430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 0x7f4d741430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 -- 172.21.15.90:0/2632765437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d74154f50 con 0x7f4d7415e4e0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d72000640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 0x7f4d741430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d73400640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0b0040 tx=0x7f4d6c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0b2050 con 0x7f4d7415e4e0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6c0b3040 con 0x7f4d7415e4e0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c0ad650 con 0x7f4d7415e4e0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d740caa70 con 0x7f4d7415e4e0 2024-03-31T15:54:35.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d740cb030 con 0x7f4d7415e4e0 2024-03-31T15:54:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.546+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d740685f0 con 0x7f4d7415e4e0 2024-03-31T15:54:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.550+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4d6c0c0020 con 0x7f4d7415e4e0 2024-03-31T15:54:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.550+0000 7f4d6aa00640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 0x7f4d5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.550+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4d6c1215a0 con 0x7f4d7415e4e0 2024-03-31T15:54:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.550+0000 7f4d72a00640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 0x7f4d5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.550+0000 7f4d72a00640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 0x7f4d5406a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009ac0 tx=0x7f4d64009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.554+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4d6c016520 con 0x7f4d7415e4e0 2024-03-31T15:54:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4d740cec50 con 0x7f4d7415e4e0 2024-03-31T15:54:35.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d6aa00640 1 -- 172.21.15.90:0/2632765437 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f4d6c0f1760 con 0x7f4d7415e4e0 2024-03-31T15:54:35.707 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 msgr2=0x7f4d5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 0x7f4d5406a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009ac0 tx=0x7f4d64009190 comp rx=0 tx=0).stop 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 msgr2=0x7f4d741435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.710+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0b0040 tx=0x7f4d6c019640 comp rx=0 tx=0).stop 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 shutdown_connections 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4d54068380 0x7f4d5406a840 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d7415e4e0 0x7f4d741435f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d740cdf20 0x7f4d741430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 --2- 172.21.15.90:0/2632765437 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4d7406ce50 0x7f4d740ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 >> 172.21.15.90:0/2632765437 conn(0x7f4d740d1f20 msgr2=0x7f4d74157550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 shutdown_connections 2024-03-31T15:54:35.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.714+0000 7f4d79800640 1 -- 172.21.15.90:0/2632765437 wait complete. 2024-03-31T15:54:35.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:35.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.873+0000 7f6a60000640 1 Processor -- start 2024-03-31T15:54:35.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.873+0000 7f6a60000640 1 -- start start 2024-03-31T15:54:35.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a60000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a5806d020 0x7f6a580ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a60000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a60000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5815e8c0 con 0x7f6a5815e4e0 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a60000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6a580d0ff0 con 0x7f6a5806d020 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a60000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6a580d12d0 con 0x7f6a5806d490 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a5806d020 0x7f6a580ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5e000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5d600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6a5815e4e0 0x7f6a5806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:34582/0 (socket says 172.21.15.90:34582) 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6a5806d020 0x7f6a580ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:49976/0 (socket says 172.21.15.90:49976) 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5e000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50054/0 (socket says 172.21.15.90:50054) 2024-03-31T15:54:35.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5d600640 1 -- 172.21.15.90:0/228030399 learned_addr learned my addr 172.21.15.90:0/228030399 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 -- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 msgr2=0x7f6a580cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 -- 172.21.15.90:0/228030399 >> v1:172.21.15.102:6789/0 conn(0x7f6a5815e4e0 legacy=0x7f6a5806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5cc00640 1 -- 172.21.15.90:0/228030399 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580cc710 con 0x7f6a5806d020 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a5e000640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:35.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.877+0000 7f6a57e00640 1 -- 172.21.15.90:0/228030399 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531194605 0 0) 0x7f6a5815e8c0 con 0x7f6a5815e4e0 2024-03-31T15:54:35.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a5cc00640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580ccf10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a440097f0 tx=0x7f6a4401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=932acdbff1ab1c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a57e00640 1 -- 172.21.15.90:0/228030399 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440b0070 con 0x7f6a5806d020 2024-03-31T15:54:35.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a57e00640 1 -- 172.21.15.90:0/228030399 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a440ab600 con 0x7f6a5806d020 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a57e00640 1 -- 172.21.15.90:0/228030399 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440ab9d0 con 0x7f6a5806d020 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 msgr2=0x7f6a580ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580ccf10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a440097f0 tx=0x7f6a4401c410 comp rx=0 tx=0).stop 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- 172.21.15.90:0/228030399 shutdown_connections 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 --2- 172.21.15.90:0/228030399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580ccf10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- 172.21.15.90:0/228030399 >> 172.21.15.90:0/228030399 conn(0x7f6a580d1f20 msgr2=0x7f6a580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:35.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- 172.21.15.90:0/228030399 shutdown_connections 2024-03-31T15:54:35.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- 172.21.15.90:0/228030399 wait complete. 2024-03-31T15:54:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 Processor -- start 2024-03-31T15:54:35.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.881+0000 7f6a60000640 1 -- start start 2024-03-31T15:54:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6a580d0d90 con 0x7f6a5815e4e0 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6a580d0c10 con 0x7f6a5806d020 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6a5816b030 con 0x7f6a5806d490 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:49984/0 (socket says 172.21.15.90:49984) 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43086/0 (socket says 172.21.15.90:43086) 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 -- 172.21.15.90:0/1182412578 learned_addr learned my addr 172.21.15.90:0/1182412578 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5cc00640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 -- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 msgr2=0x7f6a580c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 -- 172.21.15.90:0/1182412578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 msgr2=0x7f6a58154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 -- 172.21.15.90:0/1182412578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580c7400 con 0x7f6a5806d020 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5e000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:35.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a5d600640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0011f0 tx=0x7f6a4c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c0074e0 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0047f0 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c007680 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a580c76e0 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a580c7ca0 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a580685f0 con 0x7f6a5806d020 2024-03-31T15:54:35.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.885+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a4c011070 con 0x7f6a5806d020 2024-03-31T15:54:35.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.889+0000 7f6a55600640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 0x7f6a3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:35.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.889+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6a4c086b90 con 0x7f6a5806d020 2024-03-31T15:54:35.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.889+0000 7f6a5cc00640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 0x7f6a3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:35.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.889+0000 7f6a5cc00640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 0x7f6a3806a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a440ad000 tx=0x7f6a440a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:35.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:35.893+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6a4c056d30 con 0x7f6a5806d020 2024-03-31T15:54:36.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.049+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6a58154650 con 0x7f6a5806d020 2024-03-31T15:54:36.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.049+0000 7f6a55600640 1 -- 172.21.15.90:0/1182412578 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6a4c056630 con 0x7f6a5806d020 2024-03-31T15:54:36.046 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 msgr2=0x7f6a3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 0x7f6a3806a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a440ad000 tx=0x7f6a440a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 msgr2=0x7f6a580cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c0011f0 tx=0x7f6a4c00c8f0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 shutdown_connections 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6a38068380 0x7f6a3806a840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a5815e4e0 0x7f6a58154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6a5806d490 0x7f6a580c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 --2- 172.21.15.90:0/1182412578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6a5806d020 0x7f6a580cb210 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 >> 172.21.15.90:0/1182412578 conn(0x7f6a580d1f20 msgr2=0x7f6a5815d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 shutdown_connections 2024-03-31T15:54:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.053+0000 7f6a60000640 1 -- 172.21.15.90:0/1182412578 wait complete. 2024-03-31T15:54:36.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:36.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.205+0000 7f6926600640 1 Processor -- start 2024-03-31T15:54:36.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.205+0000 7f6926600640 1 -- start start 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f6926600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f692015c120 0x7f692015fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f6926600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69200d2680 0x7f69200b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f6926600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69200d2b10 con 0x7f69200b2c20 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f6926600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6920160bf0 con 0x7f692015c120 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f6926600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6920160350 con 0x7f69200d2680 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f692015c120 0x7f692015fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f692015c120 0x7f692015fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43102/0 (socket says 172.21.15.90:43102) 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 -- 172.21.15.90:0/1023873046 learned_addr learned my addr 172.21.15.90:0/1023873046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:36.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691ea00640 1 --2- 172.21.15.90:0/1023873046 >> v2:172.21.15.102:3300/0 conn(0x7f69200d2680 0x7f69200b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 -- 172.21.15.90:0/1023873046 >> v2:172.21.15.102:3300/0 conn(0x7f69200d2680 msgr2=0x7f69200b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:36.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 --2- 172.21.15.90:0/1023873046 >> v2:172.21.15.102:3300/0 conn(0x7f69200d2680 0x7f69200b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 -- 172.21.15.90:0/1023873046 >> v1:172.21.15.102:6789/0 conn(0x7f69200b2c20 legacy=0x7f69200cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.209+0000 7f691f400640 1 -- 172.21.15.90:0/1023873046 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6920160630 con 0x7f692015c120 2024-03-31T15:54:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f691f400640 1 --2- 172.21.15.90:0/1023873046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692015c120 0x7f692015fd00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f692006ed50 tx=0x7f691001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6320b926e635e68d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f691e000640 1 -- 172.21.15.90:0/1023873046 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69100ab660 con 0x7f692015c120 2024-03-31T15:54:36.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f691e000640 1 -- 172.21.15.90:0/1023873046 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69100b0070 con 0x7f692015c120 2024-03-31T15:54:36.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f691e000640 1 -- 172.21.15.90:0/1023873046 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69100aba20 con 0x7f692015c120 2024-03-31T15:54:36.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 -- 172.21.15.90:0/1023873046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692015c120 msgr2=0x7f692015fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 --2- 172.21.15.90:0/1023873046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692015c120 0x7f692015fd00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f692006ed50 tx=0x7f691001c460 comp rx=0 tx=0).stop 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 -- 172.21.15.90:0/1023873046 shutdown_connections 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 --2- 172.21.15.90:0/1023873046 >> v2:172.21.15.102:3300/0 conn(0x7f69200d2680 0x7f69200b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 --2- 172.21.15.90:0/1023873046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f692015c120 0x7f692015fd00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 -- 172.21.15.90:0/1023873046 >> 172.21.15.90:0/1023873046 conn(0x7f692009aa40 msgr2=0x7f692009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 -- 172.21.15.90:0/1023873046 shutdown_connections 2024-03-31T15:54:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.213+0000 7f6926600640 1 -- 172.21.15.90:0/1023873046 wait complete. 2024-03-31T15:54:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 Processor -- start 2024-03-31T15:54:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- start start 2024-03-31T15:54:36.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69200b2c20 0x7f6920159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 0x7f69200af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f69201691c0 con 0x7f69200d2680 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6920169040 con 0x7f69200b2c20 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6920169340 con 0x7f692015c120 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43116/0 (socket says 172.21.15.90:43116) 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 -- 172.21.15.90:0/544573628 learned_addr learned my addr 172.21.15.90:0/544573628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691f400640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69200b2c20 0x7f6920159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691fe00640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 0x7f69200af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 -- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 msgr2=0x7f69200af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 0x7f69200af6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 -- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69200b2c20 msgr2=0x7f6920159fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69200b2c20 0x7f6920159fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 -- 172.21.15.90:0/544573628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6920156b70 con 0x7f69200d2680 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691fe00640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 0x7f69200af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f691ea00640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f690c00e8a0 tx=0x7f690c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c017070 con 0x7f69200d2680 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f690c00c940 con 0x7f69200d2680 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c00ccf0 con 0x7f69200d2680 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6920156e50 con 0x7f69200d2680 2024-03-31T15:54:36.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.217+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6920157410 con 0x7f69200d2680 2024-03-31T15:54:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.221+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69200686d0 con 0x7f69200d2680 2024-03-31T15:54:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.221+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f690c01a070 con 0x7f69200d2680 2024-03-31T15:54:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.221+0000 7f6917400640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 0x7f690006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.221+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f690c086f70 con 0x7f69200d2680 2024-03-31T15:54:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.221+0000 7f691f400640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 0x7f690006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.225+0000 7f691f400640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 0x7f690006a840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69100af000 tx=0x7f69100a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.225+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f690c056ae0 con 0x7f69200d2680 2024-03-31T15:54:36.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.385+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f692015ba50 con 0x7f69200d2680 2024-03-31T15:54:36.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.385+0000 7f6917400640 1 -- 172.21.15.90:0/544573628 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f690c05ade0 con 0x7f69200d2680 2024-03-31T15:54:36.384 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:36.384 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 msgr2=0x7f690006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 0x7f690006a840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f69100af000 tx=0x7f69100a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 msgr2=0x7f69200af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f690c00e8a0 tx=0x7f690c00eca0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 shutdown_connections 2024-03-31T15:54:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6900068380 0x7f690006a840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f692015c120 0x7f69200af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f69200d2680 0x7f69200af170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 --2- 172.21.15.90:0/544573628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69200b2c20 0x7f6920159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 >> 172.21.15.90:0/544573628 conn(0x7f692009aa40 msgr2=0x7f69200704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 shutdown_connections 2024-03-31T15:54:36.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.389+0000 7f6926600640 1 -- 172.21.15.90:0/544573628 wait complete. 2024-03-31T15:54:36.406 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:36.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 Processor -- start 2024-03-31T15:54:36.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 -- start start 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd48c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd48c15e8c0 con 0x7fd48c06d460 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd48c0d0ff0 con 0x7fd48c15e4e0 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd492400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd48c0d12d0 con 0x7fd48c06d020 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd48c06d460 0x7fd48c0cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38214/0 (socket says 172.21.15.90:38214) 2024-03-31T15:54:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48be00640 1 -- 172.21.15.90:0/1111269086 learned_addr learned my addr 172.21.15.90:0/1111269086 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102673675 0 0) 0x7fd48c15e8c0 con 0x7fd48c06d460 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48aa00640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd48c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd470003530 con 0x7fd48c06d460 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2127270819 0 0) 0x7fd470003530 con 0x7fd48c06d460 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 msgr2=0x7fd48c0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd48c0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 >> v1:172.21.15.102:6789/0 conn(0x7fd48c15e4e0 legacy=0x7fd48c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48c0cc710 con 0x7fd48c06d460 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fd484001a00 con 0x7fd48c06d460 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 >> v1:172.21.15.90:6789/0 conn(0x7fd48c06d460 legacy=0x7fd48c0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd470003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd470004810 0x7fd470004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd470005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c0d12d0 con 0x7fd4700039f0 2024-03-31T15:54:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd47000bb30 con 0x7fd470004810 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.549+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd47000bd30 con 0x7fd48c06d020 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd470003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48aa00640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd470004810 0x7fd470004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48be00640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd470005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 -- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 msgr2=0x7fd470005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c06d020 0x7fd470005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 -- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd470004810 msgr2=0x7fd470004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd470004810 0x7fd470004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 -- 172.21.15.90:0/1111269086 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd470003530 con 0x7fd4700039f0 2024-03-31T15:54:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48b400640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd470003e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd47c001780 tx=0x7fd48c0ce370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0a8b678c64fe07 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0013a0 con 0x7fd4700039f0 2024-03-31T15:54:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47c0052f0 con 0x7fd4700039f0 2024-03-31T15:54:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.553+0000 7fd48a000640 1 -- 172.21.15.90:0/1111269086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c003840 con 0x7fd4700039f0 2024-03-31T15:54:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- 172.21.15.90:0/1111269086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 msgr2=0x7fd470003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 --2- 172.21.15.90:0/1111269086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd470003e10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd47c001780 tx=0x7fd48c0ce370 comp rx=0 tx=0).stop 2024-03-31T15:54:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48be00640 1 -- 172.21.15.90:0/1111269086 reap_dead start 2024-03-31T15:54:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- 172.21.15.90:0/1111269086 shutdown_connections 2024-03-31T15:54:36.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- 172.21.15.90:0/1111269086 >> 172.21.15.90:0/1111269086 conn(0x7fd48c0d1f20 msgr2=0x7fd48c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- 172.21.15.90:0/1111269086 shutdown_connections 2024-03-31T15:54:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- 172.21.15.90:0/1111269086 wait complete. 2024-03-31T15:54:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 Processor -- start 2024-03-31T15:54:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- start start 2024-03-31T15:54:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 0x7fd48c0b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c0b6e30 0x7fd48c15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c0d0d90 con 0x7fd4700039f0 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c0d0c10 con 0x7fd48c0b3e80 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd492400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd48c16a030 con 0x7fd48c0b6e30 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43142/0 (socket says 172.21.15.90:43142) 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 -- 172.21.15.90:0/3088790713 learned_addr learned my addr 172.21.15.90:0/3088790713 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48be00640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c0b6e30 0x7fd48c15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48aa00640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 0x7fd48c0b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 -- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c0b6e30 msgr2=0x7fd48c15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c0b6e30 0x7fd48c15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 -- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 msgr2=0x7fd48c0b68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 0x7fd48c0b68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48b400640 1 -- 172.21.15.90:0/3088790713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48c0b5850 con 0x7fd4700039f0 2024-03-31T15:54:36.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.557+0000 7fd48aa00640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 0x7fd48c0b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd48b400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd47c001780 tx=0x7fd470005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c007a40 con 0x7fd4700039f0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47c0bb070 con 0x7fd4700039f0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c000ee0 con 0x7fd4700039f0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48c0b5b30 con 0x7fd4700039f0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48c0b60f0 con 0x7fd4700039f0 2024-03-31T15:54:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd48c0685f0 con 0x7fd4700039f0 2024-03-31T15:54:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.561+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd47c0b4040 con 0x7fd4700039f0 2024-03-31T15:54:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.565+0000 7fd483400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 0x7fd46c06a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.565+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd47c120d40 con 0x7fd4700039f0 2024-03-31T15:54:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.565+0000 7fd48aa00640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 0x7fd46c06a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.565+0000 7fd48aa00640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 0x7fd46c06a9e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0b4280 tx=0x7fd478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.569+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd47c016130 con 0x7fd4700039f0 2024-03-31T15:54:36.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.725+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd48c168790 con 0x7fd4700039f0 2024-03-31T15:54:36.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.725+0000 7fd483400640 1 -- 172.21.15.90:0/3088790713 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fd47c0f0f00 con 0x7fd4700039f0 2024-03-31T15:54:36.723 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 msgr2=0x7fd46c06a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 0x7fd46c06a9e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0b4280 tx=0x7fd478008040 comp rx=0 tx=0).stop 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 msgr2=0x7fd48c0b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd47c001780 tx=0x7fd470005840 comp rx=0 tx=0).stop 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 shutdown_connections 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd46c068520 0x7fd46c06a9e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd48c0b6e30 0x7fd48c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd48c0b3e80 0x7fd48c0b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 --2- 172.21.15.90:0/3088790713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4700039f0 0x7fd48c0b3940 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 >> 172.21.15.90:0/3088790713 conn(0x7fd48c0d1f20 msgr2=0x7fd48c15d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 shutdown_connections 2024-03-31T15:54:36.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.729+0000 7fd492400640 1 -- 172.21.15.90:0/3088790713 wait complete. 2024-03-31T15:54:36.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:36.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 Processor -- start 2024-03-31T15:54:36.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 -- start start 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f876415b040 0x7f876415b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8764071940 0x7f8764071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8764072a60 con 0x7f87640713f0 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f87640b4a60 con 0x7f876415b040 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f876ba00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8764072d40 con 0x7f8764071940 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f876415b040 0x7f876415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f876415b040 0x7f876415b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43144/0 (socket says 172.21.15.90:43144) 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 -- 172.21.15.90:0/2946833674 learned_addr learned my addr 172.21.15.90:0/2946833674 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8763e00640 1 --2- 172.21.15.90:0/2946833674 >> v2:172.21.15.102:3300/0 conn(0x7f8764071940 0x7f8764071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 -- 172.21.15.90:0/2946833674 >> v2:172.21.15.102:3300/0 conn(0x7f8764071940 msgr2=0x7f8764071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 --2- 172.21.15.90:0/2946833674 >> v2:172.21.15.102:3300/0 conn(0x7f8764071940 0x7f8764071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 -- 172.21.15.90:0/2946833674 >> v1:172.21.15.102:6789/0 conn(0x7f87640713f0 legacy=0x7f87640ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.889+0000 7f8769000640 1 -- 172.21.15.90:0/2946833674 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87640aeac0 con 0x7f876415b040 2024-03-31T15:54:36.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.893+0000 7f8769000640 1 --2- 172.21.15.90:0/2946833674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f876415b040 0x7f876415b420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f876409e0a0 tx=0x7f875801e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71975355459bf81 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8763400640 1 -- 172.21.15.90:0/2946833674 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580ab680 con 0x7f876415b040 2024-03-31T15:54:36.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8763400640 1 -- 172.21.15.90:0/2946833674 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87580ab820 con 0x7f876415b040 2024-03-31T15:54:36.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8763400640 1 -- 172.21.15.90:0/2946833674 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580abbf0 con 0x7f876415b040 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- 172.21.15.90:0/2946833674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f876415b040 msgr2=0x7f876415b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- 172.21.15.90:0/2946833674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f876415b040 0x7f876415b420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f876409e0a0 tx=0x7f875801e6a0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- 172.21.15.90:0/2946833674 shutdown_connections 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- 172.21.15.90:0/2946833674 >> v2:172.21.15.102:3300/0 conn(0x7f8764071940 0x7f8764071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- 172.21.15.90:0/2946833674 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f876415b040 0x7f876415b420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- 172.21.15.90:0/2946833674 >> 172.21.15.90:0/2946833674 conn(0x7f87640b18b0 msgr2=0x7f87640b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- 172.21.15.90:0/2946833674 shutdown_connections 2024-03-31T15:54:36.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- 172.21.15.90:0/2946833674 wait complete. 2024-03-31T15:54:36.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 Processor -- start 2024-03-31T15:54:36.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- start start 2024-03-31T15:54:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8764071940 0x7f8764147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 0x7f87641f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f876406dc00 con 0x7f87640713f0 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f876406da80 con 0x7f876415b040 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f876ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f876406dd80 con 0x7f8764071940 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8769000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8769000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43148/0 (socket says 172.21.15.90:43148) 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8769000640 1 -- 172.21.15.90:0/2500340084 learned_addr learned my addr 172.21.15.90:0/2500340084 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8763e00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8764071940 0x7f8764147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.897+0000 7f8769a00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 0x7f87641f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 -- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8764071940 msgr2=0x7f8764147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8764071940 0x7f8764147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 -- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 msgr2=0x7f87641f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 0x7f87641f7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 -- 172.21.15.90:0/2500340084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87641f73c0 con 0x7f87640713f0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769a00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 0x7f87641f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8769000640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f875801b2b0 tx=0x7f875801b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b3800 con 0x7f87640713f0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87580b39a0 con 0x7f87640713f0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b3d50 con 0x7f87640713f0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8764158f50 con 0x7f87640713f0 2024-03-31T15:54:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87641f7660 con 0x7f87640713f0 2024-03-31T15:54:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8764075880 con 0x7f87640713f0 2024-03-31T15:54:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.901+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87580a9070 con 0x7f87640713f0 2024-03-31T15:54:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.905+0000 7f8760c00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 0x7f874406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.905+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8758120c70 con 0x7f87640713f0 2024-03-31T15:54:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.905+0000 7f8763e00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 0x7f874406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.905+0000 7f8763e00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 0x7f874406a870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8764146dd0 tx=0x7f8750008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:36.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:36.909+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f87580c2030 con 0x7f87640713f0 2024-03-31T15:54:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.065+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f87641f71b0 con 0x7f87640713f0 2024-03-31T15:54:37.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.065+0000 7f8760c00640 1 -- 172.21.15.90:0/2500340084 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f87580af020 con 0x7f87640713f0 2024-03-31T15:54:37.062 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:37.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 msgr2=0x7f874406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 0x7f874406a870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8764146dd0 tx=0x7f8750008040 comp rx=0 tx=0).stop 2024-03-31T15:54:37.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 msgr2=0x7f87641588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f875801b2b0 tx=0x7f875801b440 comp rx=0 tx=0).stop 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 shutdown_connections 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87440683b0 0x7f874406a870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f876415b040 0x7f87641f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8764071940 0x7f8764147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 --2- 172.21.15.90:0/2500340084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87640713f0 0x7f87641588c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 >> 172.21.15.90:0/2500340084 conn(0x7f87640b18b0 msgr2=0x7f876406e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 shutdown_connections 2024-03-31T15:54:37.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.069+0000 7f876ba00640 1 -- 172.21.15.90:0/2500340084 wait complete. 2024-03-31T15:54:37.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.225+0000 7fc897800640 1 Processor -- start 2024-03-31T15:54:37.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.225+0000 7fc897800640 1 -- start start 2024-03-31T15:54:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc897800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc89015c290 0x7fc8900ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc897800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc89006e9c0 con 0x7fc8900af210 2024-03-31T15:54:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc897800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc89006de40 con 0x7fc8900ad3d0 2024-03-31T15:54:37.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc897800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc89006e120 con 0x7fc89015c290 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc895800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc8900af210 0x7fc890071750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38220/0 (socket says 172.21.15.90:38220) 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc895800640 1 -- 172.21.15.90:0/2483309997 learned_addr learned my addr 172.21.15.90:0/2483309997 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc894e00640 1 --2- 172.21.15.90:0/2483309997 >> v2:172.21.15.102:3300/0 conn(0x7fc89015c290 0x7fc8900ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc88f400640 1 -- 172.21.15.90:0/2483309997 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27516428 0 0) 0x7fc89006e9c0 con 0x7fc8900af210 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc88f400640 1 -- 172.21.15.90:0/2483309997 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc878003530 con 0x7fc8900af210 2024-03-31T15:54:37.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc894e00640 1 -- 172.21.15.90:0/2483309997 >> v1:172.21.15.102:6789/0 conn(0x7fc8900ad3d0 legacy=0x7fc8900aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc894e00640 1 -- 172.21.15.90:0/2483309997 >> v1:172.21.15.90:6789/0 conn(0x7fc8900af210 legacy=0x7fc890071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.229+0000 7fc894e00640 1 -- 172.21.15.90:0/2483309997 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89006e400 con 0x7fc89015c290 2024-03-31T15:54:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc894e00640 1 --2- 172.21.15.90:0/2483309997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900ace90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8840097c0 tx=0x7fc88401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ab2dc8a3ec9747 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc88f400640 1 -- 172.21.15.90:0/2483309997 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8840b0070 con 0x7fc89015c290 2024-03-31T15:54:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc88f400640 1 -- 172.21.15.90:0/2483309997 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8840ab600 con 0x7fc89015c290 2024-03-31T15:54:37.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc88f400640 1 -- 172.21.15.90:0/2483309997 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8840ab9d0 con 0x7fc89015c290 2024-03-31T15:54:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- 172.21.15.90:0/2483309997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 msgr2=0x7fc8900ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 --2- 172.21.15.90:0/2483309997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900ace90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8840097c0 tx=0x7fc88401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- 172.21.15.90:0/2483309997 shutdown_connections 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 --2- 172.21.15.90:0/2483309997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900ace90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- 172.21.15.90:0/2483309997 >> 172.21.15.90:0/2483309997 conn(0x7fc8900b18b0 msgr2=0x7fc8900b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- 172.21.15.90:0/2483309997 shutdown_connections 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- 172.21.15.90:0/2483309997 wait complete. 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 Processor -- start 2024-03-31T15:54:37.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- start start 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8900ad3d0 0x7fc890153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc890072a00 con 0x7fc8900af210 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc890072880 con 0x7fc89015c290 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc897800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc89006dc00 con 0x7fc8900ad3d0 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.233+0000 7fc88fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43156/0 (socket says 172.21.15.90:43156) 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 -- 172.21.15.90:0/2274726715 learned_addr learned my addr 172.21.15.90:0/2274726715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc894e00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8900ad3d0 0x7fc890153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc895800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 -- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8900ad3d0 msgr2=0x7fc890153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8900ad3d0 0x7fc890153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 -- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 msgr2=0x7fc8900b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 -- 172.21.15.90:0/2274726715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8900b2de0 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88fe00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c370 tx=0x7fc87c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0052d0 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87c005470 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8900b30c0 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c007260 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8900b3680 con 0x7fc8900af210 2024-03-31T15:54:37.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc890075880 con 0x7fc8900af210 2024-03-31T15:54:37.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.237+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc87c007400 con 0x7fc8900af210 2024-03-31T15:54:37.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.241+0000 7fc88cc00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 0x7fc87006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.241+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc87c086180 con 0x7fc8900af210 2024-03-31T15:54:37.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.241+0000 7fc894e00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 0x7fc87006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.241+0000 7fc894e00640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 0x7fc87006a840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8840ad000 tx=0x7fc884008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.245+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc87c055d00 con 0x7fc8900af210 2024-03-31T15:54:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8900b3850 con 0x7fc8900af210 2024-03-31T15:54:37.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc88cc00640 1 -- 172.21.15.90:0/2274726715 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc87c05a000 con 0x7fc8900af210 2024-03-31T15:54:37.413 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:37.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 msgr2=0x7fc87006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 0x7fc87006a840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8840ad000 tx=0x7fc884008040 comp rx=0 tx=0).stop 2024-03-31T15:54:37.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 msgr2=0x7fc8901671f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c370 tx=0x7fc87c00c830 comp rx=0 tx=0).stop 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 shutdown_connections 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc870068380 0x7fc87006a840 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc89015c290 0x7fc8900b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8900af210 0x7fc8901671f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 --2- 172.21.15.90:0/2274726715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8900ad3d0 0x7fc890153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 >> 172.21.15.90:0/2274726715 conn(0x7fc8900b18b0 msgr2=0x7fc8901575c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 shutdown_connections 2024-03-31T15:54:37.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.417+0000 7fc897800640 1 -- 172.21.15.90:0/2274726715 wait complete. 2024-03-31T15:54:37.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:37.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.581+0000 7f6ea8a00640 1 Processor -- start 2024-03-31T15:54:37.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.581+0000 7f6ea8a00640 1 -- start start 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea8a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ea4070840 0x7f6ea40add80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea8a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ea40b3410 0x7f6ea415fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea8a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea40ae880 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea8a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea415d080 con 0x7f6ea4070840 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea8a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea40cc730 con 0x7f6ea40b3410 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6e9be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6ea40ae2c0 0x7f6ea40b2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38230/0 (socket says 172.21.15.90:38230) 2024-03-31T15:54:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6e9be00640 1 -- 172.21.15.90:0/403874727 learned_addr learned my addr 172.21.15.90:0/403874727 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/403874727 >> v2:172.21.15.90:3300/0 conn(0x7f6ea4070840 0x7f6ea40add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea2800640 1 --2- 172.21.15.90:0/403874727 >> v2:172.21.15.102:3300/0 conn(0x7f6ea40b3410 0x7f6ea415fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1400640 1 -- 172.21.15.90:0/403874727 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922860940 0 0) 0x7f6ea40ae880 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1e00640 1 -- 172.21.15.90:0/403874727 >> v2:172.21.15.102:3300/0 conn(0x7f6ea40b3410 msgr2=0x7f6ea415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/403874727 >> v2:172.21.15.102:3300/0 conn(0x7f6ea40b3410 0x7f6ea415fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1e00640 1 -- 172.21.15.90:0/403874727 >> v1:172.21.15.90:6789/0 conn(0x7f6ea40ae2c0 legacy=0x7f6ea40b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.585+0000 7f6ea1e00640 1 -- 172.21.15.90:0/403874727 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea415c750 con 0x7f6ea4070840 2024-03-31T15:54:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.589+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/403874727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea4070840 0x7f6ea40add80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4156b40 tx=0x7f6e9401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3923ff7df02695b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.589+0000 7f6ea1400640 1 -- 172.21.15.90:0/403874727 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e940b0070 con 0x7f6ea4070840 2024-03-31T15:54:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.589+0000 7f6ea1400640 1 -- 172.21.15.90:0/403874727 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e940ab600 con 0x7f6ea4070840 2024-03-31T15:54:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.589+0000 7f6ea1400640 1 -- 172.21.15.90:0/403874727 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e940ab9d0 con 0x7f6ea4070840 2024-03-31T15:54:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- 172.21.15.90:0/403874727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea4070840 msgr2=0x7f6ea40add80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/403874727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea4070840 0x7f6ea40add80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4156b40 tx=0x7f6e9401c480 comp rx=0 tx=0).stop 2024-03-31T15:54:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- 172.21.15.90:0/403874727 shutdown_connections 2024-03-31T15:54:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/403874727 >> v2:172.21.15.102:3300/0 conn(0x7f6ea40b3410 0x7f6ea415fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/403874727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea4070840 0x7f6ea40add80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- 172.21.15.90:0/403874727 >> 172.21.15.90:0/403874727 conn(0x7f6ea40cde90 msgr2=0x7f6ea40ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- 172.21.15.90:0/403874727 shutdown_connections 2024-03-31T15:54:37.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- 172.21.15.90:0/403874727 wait complete. 2024-03-31T15:54:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 Processor -- start 2024-03-31T15:54:37.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.593+0000 7f6ea8a00640 1 -- start start 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4070840 0x7f6ea40c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 0x7f6ea4156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4070da0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4070c20 con 0x7f6ea40b3410 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea40cb720 con 0x7f6ea4070840 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43174/0 (socket says 172.21.15.90:43174) 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 -- 172.21.15.90:0/3672316102 learned_addr learned my addr 172.21.15.90:0/3672316102 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4070840 0x7f6ea40c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea2800640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 0x7f6ea4156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 -- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4070840 msgr2=0x7f6ea40c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4070840 0x7f6ea40c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 -- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 msgr2=0x7f6ea4156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 0x7f6ea4156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 -- 172.21.15.90:0/3672316102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea40b4ae0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea2800640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 0x7f6ea4156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9be00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c550 tx=0x7f6e8c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c017070 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8c005d10 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c007330 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea40c8630 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea4154450 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea40685f0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.597+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6e8c01a070 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.601+0000 7f6e9a000640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.601+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6e8c0860e0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.601+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.601+0000 7f6ea1e00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6e940ad000 tx=0x7f6e94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.605+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6e8c055ce0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6ea40b6d90 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6e9a000640 1 -- 172.21.15.90:0/3672316102 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6e8c059fe0 con 0x7f6ea40ae2c0 2024-03-31T15:54:37.758 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 msgr2=0x7f6e8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6e940ad000 tx=0x7f6e94008040 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 msgr2=0x7f6ea4153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c550 tx=0x7f6e8c00ca10 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 shutdown_connections 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea40b3410 0x7f6ea4156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ae2c0 0x7f6ea4153f10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 --2- 172.21.15.90:0/3672316102 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4070840 0x7f6ea40c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 >> 172.21.15.90:0/3672316102 conn(0x7f6ea40cde90 msgr2=0x7f6ea40d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 shutdown_connections 2024-03-31T15:54:37.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.761+0000 7f6ea8a00640 1 -- 172.21.15.90:0/3672316102 wait complete. 2024-03-31T15:54:37.778 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 Processor -- start 2024-03-31T15:54:37.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 -- start start 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e8815a7c0 0x7f4e88071a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e88071f80 0x7f4e88072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e8806e050 con 0x7f4e8815e1b0 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8806e330 con 0x7f4e88071f80 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e8cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4e880b2c20 con 0x7f4e8815a7c0 2024-03-31T15:54:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e86800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4e8815e1b0 0x7f4e8815e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38246/0 (socket says 172.21.15.90:38246) 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e85400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e88071f80 0x7f4e88072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.917+0000 7f4e86800640 1 -- 172.21.15.90:0/436457485 learned_addr learned my addr 172.21.15.90:0/436457485 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85e00640 1 --2- 172.21.15.90:0/436457485 >> v2:172.21.15.102:3300/0 conn(0x7f4e8815a7c0 0x7f4e88071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85400640 1 -- 172.21.15.90:0/436457485 >> v2:172.21.15.102:3300/0 conn(0x7f4e8815a7c0 msgr2=0x7f4e88071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85400640 1 --2- 172.21.15.90:0/436457485 >> v2:172.21.15.102:3300/0 conn(0x7f4e8815a7c0 0x7f4e88071a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85400640 1 -- 172.21.15.90:0/436457485 >> v1:172.21.15.90:6789/0 conn(0x7f4e8815e1b0 legacy=0x7f4e8815e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85400640 1 -- 172.21.15.90:0/436457485 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e880b2f00 con 0x7f4e88071f80 2024-03-31T15:54:37.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e84a00640 1 -- 172.21.15.90:0/436457485 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990745691 0 0) 0x7f4e8806e050 con 0x7f4e8815e1b0 2024-03-31T15:54:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e85400640 1 --2- 172.21.15.90:0/436457485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e88071f80 0x7f4e88072400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806c4c0 tx=0x7f4e7401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569b77ab972eb982 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e84a00640 1 -- 172.21.15.90:0/436457485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740b0070 con 0x7f4e88071f80 2024-03-31T15:54:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e84a00640 1 -- 172.21.15.90:0/436457485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e740ab400 con 0x7f4e88071f80 2024-03-31T15:54:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.921+0000 7f4e84a00640 1 -- 172.21.15.90:0/436457485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740ab7d0 con 0x7f4e88071f80 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- 172.21.15.90:0/436457485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e88071f80 msgr2=0x7f4e88072400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/436457485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e88071f80 0x7f4e88072400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4e8806c4c0 tx=0x7f4e7401c3f0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- 172.21.15.90:0/436457485 shutdown_connections 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/436457485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e88071f80 0x7f4e88072400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/436457485 >> v2:172.21.15.102:3300/0 conn(0x7f4e8815a7c0 0x7f4e88071a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- 172.21.15.90:0/436457485 >> 172.21.15.90:0/436457485 conn(0x7f4e880b1c80 msgr2=0x7f4e880b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- 172.21.15.90:0/436457485 shutdown_connections 2024-03-31T15:54:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- 172.21.15.90:0/436457485 wait complete. 2024-03-31T15:54:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 Processor -- start 2024-03-31T15:54:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- start start 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e88071f80 0x7f4e881438b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e8815e1b0 0x7f4e881318a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88072d20 con 0x7f4e8815a7c0 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88072ba0 con 0x7f4e8815e1b0 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e8cc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4e880b4430 con 0x7f4e88071f80 2024-03-31T15:54:37.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43194/0 (socket says 172.21.15.90:43194) 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 -- 172.21.15.90:0/1669477382 learned_addr learned my addr 172.21.15.90:0/1669477382 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 -- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e88071f80 msgr2=0x7f4e881438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e88071f80 0x7f4e881438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 -- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e8815e1b0 msgr2=0x7f4e881318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e8815e1b0 0x7f4e881318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 -- 172.21.15.90:0/1669477382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e88131de0 con 0x7f4e8815a7c0 2024-03-31T15:54:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e85400640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4e74019000 tx=0x7f4e7401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740b0050 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.925+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e740bc070 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740b7670 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e88132090 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e88132650 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e880686d0 con 0x7f4e8815a7c0 2024-03-31T15:54:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e740b7810 con 0x7f4e8815a7c0 2024-03-31T15:54:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e7e000640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4e74121490 con 0x7f4e8815a7c0 2024-03-31T15:54:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e85e00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.929+0000 7f4e85e00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 0x7f4e6806a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006aa0 tx=0x7f4e78006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:37.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:37.937+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e74016070 con 0x7f4e8815a7c0 2024-03-31T15:54:38.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.097+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e88073570 con 0x7f4e8815a7c0 2024-03-31T15:54:38.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.097+0000 7f4e7e000640 1 -- 172.21.15.90:0/1669477382 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f4e740f1760 con 0x7f4e8815a7c0 2024-03-31T15:54:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:38.095 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 msgr2=0x7f4e6806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 0x7f4e6806a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4e78006aa0 tx=0x7f4e78006a30 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 msgr2=0x7f4e88131360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4e74019000 tx=0x7f4e7401ac30 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 shutdown_connections 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4e68068380 0x7f4e6806a840 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e8815e1b0 0x7f4e881318a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e8815a7c0 0x7f4e88131360 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 --2- 172.21.15.90:0/1669477382 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e88071f80 0x7f4e881438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 >> 172.21.15.90:0/1669477382 conn(0x7f4e880b1c80 msgr2=0x7f4e880ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 shutdown_connections 2024-03-31T15:54:38.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.101+0000 7f4e8cc00640 1 -- 172.21.15.90:0/1669477382 wait complete. 2024-03-31T15:54:38.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:38.115 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:38.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.257+0000 7f60be400640 1 Processor -- start 2024-03-31T15:54:38.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.257+0000 7f60be400640 1 -- start start 2024-03-31T15:54:38.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60b8070840 0x7f60b80add80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 0x7f60b80b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 0x7f60b806b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f60b80b3690 con 0x7f60b8070840 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f60b815d080 con 0x7f60b80b3240 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60be400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f60b80ae880 con 0x7f60b80ae2c0 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60b8070840 0x7f60b80add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f60b8070840 0x7f60b80add80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43202/0 (socket says 172.21.15.90:43202) 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 -- 172.21.15.90:0/2421359843 learned_addr learned my addr 172.21.15.90:0/2421359843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60afe00640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 0x7f60b80b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 -- 172.21.15.90:0/2421359843 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 msgr2=0x7f60b80b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7e00640 1 --2- 172.21.15.90:0/2421359843 >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 0x7f60b806b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 0x7f60b80b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 -- 172.21.15.90:0/2421359843 >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 msgr2=0x7f60b806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 --2- 172.21.15.90:0/2421359843 >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 0x7f60b806b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7400640 1 -- 172.21.15.90:0/2421359843 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b815c750 con 0x7f60b8070840 2024-03-31T15:54:38.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60afe00640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 0x7f60b80b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:38.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.261+0000 7f60b7e00640 1 --2- 172.21.15.90:0/2421359843 >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 0x7f60b806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:38.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60b7400640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b80add80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f60b815a0a0 tx=0x7f60a801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42537205df7910b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60b6a00640 1 -- 172.21.15.90:0/2421359843 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a80b0070 con 0x7f60b8070840 2024-03-31T15:54:38.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60b6a00640 1 -- 172.21.15.90:0/2421359843 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a80ab600 con 0x7f60b8070840 2024-03-31T15:54:38.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60b6a00640 1 -- 172.21.15.90:0/2421359843 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a80ab9d0 con 0x7f60b8070840 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- 172.21.15.90:0/2421359843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 msgr2=0x7f60b80add80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b80add80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f60b815a0a0 tx=0x7f60a801c260 comp rx=0 tx=0).stop 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- 172.21.15.90:0/2421359843 shutdown_connections 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- 172.21.15.90:0/2421359843 >> v2:172.21.15.102:3300/0 conn(0x7f60b80b3240 0x7f60b806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80ae2c0 0x7f60b80b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- 172.21.15.90:0/2421359843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b80add80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- 172.21.15.90:0/2421359843 >> 172.21.15.90:0/2421359843 conn(0x7f60b80cde90 msgr2=0x7f60b80ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- 172.21.15.90:0/2421359843 shutdown_connections 2024-03-31T15:54:38.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- 172.21.15.90:0/2421359843 wait complete. 2024-03-31T15:54:38.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 Processor -- start 2024-03-31T15:54:38.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- start start 2024-03-31T15:54:38.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60b80ae2c0 0x7f60b8155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80b3240 0x7f60b80ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f60b8070da0 con 0x7f60b8070840 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f60b8070c20 con 0x7f60b80ae2c0 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.265+0000 7f60be400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f60b815fb50 con 0x7f60b80b3240 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43204/0 (socket says 172.21.15.90:43204) 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 -- 172.21.15.90:0/1592693200 learned_addr learned my addr 172.21.15.90:0/1592693200 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7e00640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80b3240 0x7f60b80ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60afe00640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60b80ae2c0 0x7f60b8155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 -- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80b3240 msgr2=0x7f60b80ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80b3240 0x7f60b80ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 -- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60b80ae2c0 msgr2=0x7f60b8155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60b80ae2c0 0x7f60b8155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 -- 172.21.15.90:0/1592693200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b80b5960 con 0x7f60b8070840 2024-03-31T15:54:38.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60b7400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60a80ae040 tx=0x7f60a8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a80b0040 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a80b9070 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a80b48a0 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b80b5c40 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b80b6200 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b80685f0 con 0x7f60b8070840 2024-03-31T15:54:38.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.269+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f60a80ab4c0 con 0x7f60b8070840 2024-03-31T15:54:38.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.273+0000 7f60af400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 0x7f609406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.273+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f60a8120b20 con 0x7f60b8070840 2024-03-31T15:54:38.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.273+0000 7f60afe00640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 0x7f609406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.273+0000 7f60afe00640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 0x7f609406a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60b809efb0 tx=0x7f60a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.277+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f60a8016110 con 0x7f60b8070840 2024-03-31T15:54:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f60b806f970 con 0x7f60b8070840 2024-03-31T15:54:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60af400640 1 -- 172.21.15.90:0/1592693200 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f60a80f0690 con 0x7f60b8070840 2024-03-31T15:54:38.433 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 msgr2=0x7f609406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 0x7f609406a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60b809efb0 tx=0x7f60a0008040 comp rx=0 tx=0).stop 2024-03-31T15:54:38.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 msgr2=0x7f60b81419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60a80ae040 tx=0x7f60a8019000 comp rx=0 tx=0).stop 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 shutdown_connections 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6094068380 0x7f609406a840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f60b80b3240 0x7f60b80ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60b80ae2c0 0x7f60b8155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 --2- 172.21.15.90:0/1592693200 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60b8070840 0x7f60b81419b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 >> 172.21.15.90:0/1592693200 conn(0x7f60b80cde90 msgr2=0x7f60b80cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 shutdown_connections 2024-03-31T15:54:38.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.437+0000 7f60be400640 1 -- 172.21.15.90:0/1592693200 wait complete. 2024-03-31T15:54:38.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:38.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 Processor -- start 2024-03-31T15:54:38.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 -- start start 2024-03-31T15:54:38.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f700c0cfe50 0x7f700c0d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700c0b71d0 con 0x7f700c0d2c90 2024-03-31T15:54:38.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700c0b74b0 con 0x7f700c15e6a0 2024-03-31T15:54:38.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7011e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f700c0b7790 con 0x7f700c0cfe50 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f7003e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f700c0cfe50 0x7f700c0d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f700b400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f700c0d2c90 0x7f700c0b4de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38258/0 (socket says 172.21.15.90:38258) 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.597+0000 7f700b400640 1 -- 172.21.15.90:0/687023664 learned_addr learned my addr 172.21.15.90:0/687023664 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7003e00640 1 -- 172.21.15.90:0/687023664 >> v1:172.21.15.102:6789/0 conn(0x7f700c15e6a0 legacy=0x7f700c0cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7003e00640 1 -- 172.21.15.90:0/687023664 >> v1:172.21.15.90:6789/0 conn(0x7f700c0d2c90 legacy=0x7f700c0b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7003e00640 1 -- 172.21.15.90:0/687023664 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c0b7a70 con 0x7f700c0cfe50 2024-03-31T15:54:38.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7003e00640 1 --2- 172.21.15.90:0/687023664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c0cfe50 0x7f700c0d2750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40097c0 tx=0x7f6ff401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6302b1e0ffcf3bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f700a000640 1 -- 172.21.15.90:0/687023664 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b5070 con 0x7f700c0cfe50 2024-03-31T15:54:38.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f700a000640 1 -- 172.21.15.90:0/687023664 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff40a8990 con 0x7f700c0cfe50 2024-03-31T15:54:38.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f700a000640 1 -- 172.21.15.90:0/687023664 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40a8d60 con 0x7f700c0cfe50 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7011e00640 1 -- 172.21.15.90:0/687023664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c0cfe50 msgr2=0x7f700c0d2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.601+0000 7f7011e00640 1 --2- 172.21.15.90:0/687023664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c0cfe50 0x7f700c0d2750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40097c0 tx=0x7f6ff401ec90 comp rx=0 tx=0).stop 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- 172.21.15.90:0/687023664 shutdown_connections 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 --2- 172.21.15.90:0/687023664 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c0cfe50 0x7f700c0d2750 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- 172.21.15.90:0/687023664 >> 172.21.15.90:0/687023664 conn(0x7f700c09d4b0 msgr2=0x7f700c0d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- 172.21.15.90:0/687023664 shutdown_connections 2024-03-31T15:54:38.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- 172.21.15.90:0/687023664 wait complete. 2024-03-31T15:54:38.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 Processor -- start 2024-03-31T15:54:38.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- start start 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f700c0cfe50 0x7f700c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 0x7f700c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f700c0b9400 con 0x7f700c15e6a0 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f700c0b9280 con 0x7f700c0d2c90 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7011e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f700c0b9580 con 0x7f700c0cfe50 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43218/0 (socket says 172.21.15.90:43218) 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 -- 172.21.15.90:0/3657368654 learned_addr learned my addr 172.21.15.90:0/3657368654 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700aa00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f700c0cfe50 0x7f700c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7003e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 0x7f700c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 -- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f700c0cfe50 msgr2=0x7f700c159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f700c0cfe50 0x7f700c159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 -- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 msgr2=0x7f700c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 0x7f700c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 -- 172.21.15.90:0/3657368654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c0cc940 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7003e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 0x7f700c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f700b400640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f700400cbf0 tx=0x7f700400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.605+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004017070 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f700400ad40 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004012600 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700c0ccc20 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f700c0cd1e0 con 0x7f700c15e6a0 2024-03-31T15:54:38.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f700c0685f0 con 0x7f700c15e6a0 2024-03-31T15:54:38.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f700401a070 con 0x7f700c15e6a0 2024-03-31T15:54:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7003400640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 0x7f6fe806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7004086aa0 con 0x7f700c15e6a0 2024-03-31T15:54:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.609+0000 7f700aa00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 0x7f6fe806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.613+0000 7f700aa00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 0x7f6fe806a870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc000f70 tx=0x7f6ffc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.613+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7004056600 con 0x7f700c15e6a0 2024-03-31T15:54:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f700c15ef40 con 0x7f700c15e6a0 2024-03-31T15:54:38.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7003400640 1 -- 172.21.15.90:0/3657368654 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f700405a900 con 0x7f700c15e6a0 2024-03-31T15:54:38.769 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 msgr2=0x7f6fe806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 0x7f6fe806a870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc000f70 tx=0x7f6ffc009040 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 msgr2=0x7f700c1588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f700400cbf0 tx=0x7f700400a430 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 shutdown_connections 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fe80683b0 0x7f6fe806a870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f700c15e6a0 0x7f700c1588d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f700c0d2c90 0x7f700c0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 --2- 172.21.15.90:0/3657368654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f700c0cfe50 0x7f700c159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 >> 172.21.15.90:0/3657368654 conn(0x7f700c09d4b0 msgr2=0x7f700c06db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 shutdown_connections 2024-03-31T15:54:38.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.773+0000 7f7011e00640 1 -- 172.21.15.90:0/3657368654 wait complete. 2024-03-31T15:54:38.791 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:38.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 Processor -- start 2024-03-31T15:54:38.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 -- start start 2024-03-31T15:54:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8e015e4e0 0x7fc8e006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e006ce50 0x7fc8e006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc8e00ce7a0 0x7fc8e00cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e015e8c0 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e00d0ff0 con 0x7fc8e015e4e0 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e6c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00d12d0 con 0x7fc8e006ce50 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc8e00ce7a0 0x7fc8e00cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc8e00ce7a0 0x7fc8e00cec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43220/0 (socket says 172.21.15.90:43220) 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 -- 172.21.15.90:0/189194936 learned_addr learned my addr 172.21.15.90:0/189194936 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8d7e00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e006ce50 0x7fc8e006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8dfe00640 1 --2- 172.21.15.90:0/189194936 >> v2:172.21.15.102:3300/0 conn(0x7fc8e015e4e0 0x7fc8e006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 -- 172.21.15.90:0/189194936 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e006ce50 msgr2=0x7fc8e006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e006ce50 0x7fc8e006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 -- 172.21.15.90:0/189194936 >> v2:172.21.15.102:3300/0 conn(0x7fc8e015e4e0 msgr2=0x7fc8e006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 --2- 172.21.15.90:0/189194936 >> v2:172.21.15.102:3300/0 conn(0x7fc8e015e4e0 0x7fc8e006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.933+0000 7fc8e4c00640 1 -- 172.21.15.90:0/189194936 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00cc710 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e4c00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e00cec30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00d3020 tx=0x7fc8d801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553e10b4667e8fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8df400640 1 -- 172.21.15.90:0/189194936 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80a55e0 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8df400640 1 -- 172.21.15.90:0/189194936 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d80a5780 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8df400640 1 -- 172.21.15.90:0/189194936 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80a5b50 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 -- 172.21.15.90:0/189194936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e00ce7a0 msgr2=0x7fc8e00cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e00cec30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00d3020 tx=0x7fc8d801c960 comp rx=0 tx=0).stop 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 -- 172.21.15.90:0/189194936 shutdown_connections 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e00cec30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/189194936 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e006ce50 0x7fc8e006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/189194936 >> v2:172.21.15.102:3300/0 conn(0x7fc8e015e4e0 0x7fc8e006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 -- 172.21.15.90:0/189194936 >> 172.21.15.90:0/189194936 conn(0x7fc8e00d1f20 msgr2=0x7fc8e00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 -- 172.21.15.90:0/189194936 shutdown_connections 2024-03-31T15:54:38.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.937+0000 7fc8e6c00640 1 -- 172.21.15.90:0/189194936 wait complete. 2024-03-31T15:54:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 Processor -- start 2024-03-31T15:54:38.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- start start 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e0154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 0x7fc8e01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00cd090 con 0x7fc8e006ce50 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00ccf10 con 0x7fc8e00ce7a0 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00cd210 con 0x7fc8e015e4e0 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43230/0 (socket says 172.21.15.90:43230) 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 -- 172.21.15.90:0/686244916 learned_addr learned my addr 172.21.15.90:0/686244916 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e4c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 0x7fc8e01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8d7e00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e0154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 -- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 msgr2=0x7fc8e01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 0x7fc8e01552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 -- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 msgr2=0x7fc8e0154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e0154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 -- 172.21.15.90:0/686244916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00ca0e0 con 0x7fc8e006ce50 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e4c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 0x7fc8e01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8d7e00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:38.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dfe00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000d620 tx=0x7fc8d000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000b620 con 0x7fc8e006ce50 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0019070 con 0x7fc8e006ce50 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e0158620 con 0x7fc8e006ce50 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000b9e0 con 0x7fc8e006ce50 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.941+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e009c260 con 0x7fc8e006ce50 2024-03-31T15:54:38.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.945+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e00685f0 con 0x7fc8e006ce50 2024-03-31T15:54:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.945+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc8d000bb80 con 0x7fc8e006ce50 2024-03-31T15:54:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.945+0000 7fc8dcc00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 0x7fc8bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:38.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.945+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc8d00870c0 con 0x7fc8e006ce50 2024-03-31T15:54:38.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.945+0000 7fc8d7e00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 0x7fc8bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:38.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.949+0000 7fc8d7e00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 0x7fc8bc06a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0073830 tx=0x7fc8c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:38.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:38.949+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc8d0057310 con 0x7fc8e006ce50 2024-03-31T15:54:39.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.109+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8e01578b0 con 0x7fc8e006ce50 2024-03-31T15:54:39.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8dcc00640 1 -- 172.21.15.90:0/686244916 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc8d0056c10 con 0x7fc8e006ce50 2024-03-31T15:54:39.108 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:39.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 msgr2=0x7fc8bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 0x7fc8bc06a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0073830 tx=0x7fc8c8008040 comp rx=0 tx=0).stop 2024-03-31T15:54:39.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 msgr2=0x7fc8e0154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000d620 tx=0x7fc8d000dae0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 shutdown_connections 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc8bc068380 0x7fc8bc06a840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc8e015e4e0 0x7fc8e01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e00ce7a0 0x7fc8e0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 --2- 172.21.15.90:0/686244916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc8e006ce50 0x7fc8e0154680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 >> 172.21.15.90:0/686244916 conn(0x7fc8e00d1f20 msgr2=0x7fc8e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 shutdown_connections 2024-03-31T15:54:39.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.113+0000 7fc8e6c00640 1 -- 172.21.15.90:0/686244916 wait complete. 2024-03-31T15:54:39.129 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 Processor -- start 2024-03-31T15:54:39.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 -- start start 2024-03-31T15:54:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 0x7f37bc06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37bc15e8c0 con 0x7f37bc06ce50 2024-03-31T15:54:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f37bc0d0ff0 con 0x7f37bc15e4e0 2024-03-31T15:54:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37c0c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f37bc0d12d0 con 0x7f37bc0cdf20 2024-03-31T15:54:39.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37b9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 0x7f37bc06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50210/0 (socket says 172.21.15.90:50210) 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 -- 172.21.15.90:0/80267782 learned_addr learned my addr 172.21.15.90:0/80267782 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 -- 172.21.15.90:0/80267782 >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 msgr2=0x7f37bc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 --2- 172.21.15.90:0/80267782 >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 0x7f37bc06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 -- 172.21.15.90:0/80267782 >> v1:172.21.15.102:6789/0 conn(0x7f37bc06ce50 legacy=0x7f37bc0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37ba800640 1 -- 172.21.15.90:0/80267782 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc0cc710 con 0x7f37bc0cdf20 2024-03-31T15:54:39.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.273+0000 7f37b9e00640 1 --2- 172.21.15.90:0/80267782 >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 0x7f37bc06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37ba800640 1 --2- 172.21.15.90:0/80267782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009e80 tx=0x7f37b401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e54e7d77743505 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37b9400640 1 -- 172.21.15.90:0/80267782 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b40a5750 con 0x7f37bc0cdf20 2024-03-31T15:54:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37b9400640 1 -- 172.21.15.90:0/80267782 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b40a58f0 con 0x7f37bc0cdf20 2024-03-31T15:54:39.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37b9400640 1 -- 172.21.15.90:0/80267782 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b40a5cc0 con 0x7f37bc0cdf20 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 -- 172.21.15.90:0/80267782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 msgr2=0x7f37bc06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 --2- 172.21.15.90:0/80267782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f37b4009e80 tx=0x7f37b401c980 comp rx=0 tx=0).stop 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 -- 172.21.15.90:0/80267782 shutdown_connections 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 --2- 172.21.15.90:0/80267782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc06d2c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 --2- 172.21.15.90:0/80267782 >> v2:172.21.15.102:3300/0 conn(0x7f37bc15e4e0 0x7f37bc06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 -- 172.21.15.90:0/80267782 >> 172.21.15.90:0/80267782 conn(0x7f37bc0d1f20 msgr2=0x7f37bc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 -- 172.21.15.90:0/80267782 shutdown_connections 2024-03-31T15:54:39.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.277+0000 7f37c0c00640 1 -- 172.21.15.90:0/80267782 wait complete. 2024-03-31T15:54:39.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 Processor -- start 2024-03-31T15:54:39.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 -- start start 2024-03-31T15:54:39.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc15cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 0x7f37bc0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc16b1b0 con 0x7f37bc06ce50 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc16b030 con 0x7f37bc15e4e0 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f37bc16b330 con 0x7f37bc0cdf20 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43246/0 (socket says 172.21.15.90:43246) 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 -- 172.21.15.90:0/3798426235 learned_addr learned my addr 172.21.15.90:0/3798426235 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37ba800640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 0x7f37bc0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc15cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 -- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 msgr2=0x7f37bc15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc15cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 -- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 msgr2=0x7f37bc0c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 0x7f37bc0c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 -- 172.21.15.90:0/3798426235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc15b580 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37ba800640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 0x7f37bc0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b9e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00ebe0 tx=0x7f37ac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac017070 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ac00cc30 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac013470 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.281+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc15d480 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc15bb00 con 0x7f37bc06ce50 2024-03-31T15:54:39.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37bc0685f0 con 0x7f37bc06ce50 2024-03-31T15:54:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f37ac01a070 con 0x7f37bc06ce50 2024-03-31T15:54:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37b2000640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 0x7f379806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f37ac086d20 con 0x7f37bc06ce50 2024-03-31T15:54:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.285+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 0x7f379806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.289+0000 7f37b3e00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 0x7f379806a870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f37bc06cd90 tx=0x7f37a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.289+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f37ac056770 con 0x7f37bc06ce50 2024-03-31T15:54:39.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f37bc0c6ec0 con 0x7f37bc06ce50 2024-03-31T15:54:39.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37b2000640 1 -- 172.21.15.90:0/3798426235 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f37ac05aa70 con 0x7f37bc06ce50 2024-03-31T15:54:39.446 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:39.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 msgr2=0x7f379806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 0x7f379806a870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f37bc06cd90 tx=0x7f37a4009040 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 msgr2=0x7f37bc15c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00ebe0 tx=0x7f37ac00c520 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 shutdown_connections 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f37980683b0 0x7f379806a870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37bc15e4e0 0x7f37bc0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f37bc0cdf20 0x7f37bc15cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 --2- 172.21.15.90:0/3798426235 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37bc06ce50 0x7f37bc15c020 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 >> 172.21.15.90:0/3798426235 conn(0x7f37bc0d1f20 msgr2=0x7f37bc0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 shutdown_connections 2024-03-31T15:54:39.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.449+0000 7f37c0c00640 1 -- 172.21.15.90:0/3798426235 wait complete. 2024-03-31T15:54:39.467 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.605+0000 7f7aee200640 1 Processor -- start 2024-03-31T15:54:39.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.605+0000 7f7aee200640 1 -- start start 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7aee200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7aee200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ae806d5f0 0x7f7ae815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7aee200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae815f780 con 0x7f7ae815e1b0 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7aee200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae806daa0 con 0x7f7ae806d5f0 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7aee200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae81622b0 con 0x7f7ae806c7b0 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7ae7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ae806d5f0 0x7f7ae815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7adea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7adea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50222/0 (socket says 172.21.15.90:50222) 2024-03-31T15:54:39.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7adea00640 1 -- 172.21.15.90:0/3047873307 learned_addr learned my addr 172.21.15.90:0/3047873307 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7ae7e00640 1 -- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 msgr2=0x7f7ae806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7ae7e00640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7ae7e00640 1 -- 172.21.15.90:0/3047873307 >> v1:172.21.15.102:6789/0 conn(0x7f7ae815e1b0 legacy=0x7f7ae8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7ae7e00640 1 -- 172.21.15.90:0/3047873307 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae815f1c0 con 0x7f7ae806d5f0 2024-03-31T15:54:39.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.609+0000 7f7adea00640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7ae7e00640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815e590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00097c0 tx=0x7f7ae001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da9a88de44bc320 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7ae6a00640 1 -- 172.21.15.90:0/3047873307 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00b1070 con 0x7f7ae806d5f0 2024-03-31T15:54:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7ae6a00640 1 -- 172.21.15.90:0/3047873307 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae00a8050 con 0x7f7ae806d5f0 2024-03-31T15:54:39.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7ae6a00640 1 -- 172.21.15.90:0/3047873307 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00ac860 con 0x7f7ae806d5f0 2024-03-31T15:54:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806d5f0 msgr2=0x7f7ae815e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815e590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00097c0 tx=0x7f7ae001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- 172.21.15.90:0/3047873307 shutdown_connections 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815e590 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- 172.21.15.90:0/3047873307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae806c7b0 0x7f7ae806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- 172.21.15.90:0/3047873307 >> 172.21.15.90:0/3047873307 conn(0x7f7ae809aa40 msgr2=0x7f7ae809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- 172.21.15.90:0/3047873307 shutdown_connections 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- 172.21.15.90:0/3047873307 wait complete. 2024-03-31T15:54:39.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 Processor -- start 2024-03-31T15:54:39.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- start start 2024-03-31T15:54:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806c7b0 0x7f7ae80c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae815e1b0 0x7f7ae80b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae80d2940 con 0x7f7ae806d5f0 2024-03-31T15:54:39.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae80d27c0 con 0x7f7ae806c7b0 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.613+0000 7f7aee200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae80d2ac0 con 0x7f7ae815e1b0 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43256/0 (socket says 172.21.15.90:43256) 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 -- 172.21.15.90:0/1742246170 learned_addr learned my addr 172.21.15.90:0/1742246170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7ae7400640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806c7b0 0x7f7ae80c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7ae7e00640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae815e1b0 0x7f7ae80b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 -- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae815e1b0 msgr2=0x7f7ae80b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae815e1b0 0x7f7ae80b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 -- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806c7b0 msgr2=0x7f7ae80c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806c7b0 0x7f7ae80c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 -- 172.21.15.90:0/1742246170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae815d040 con 0x7f7ae806d5f0 2024-03-31T15:54:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adea00640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000e930 tx=0x7f7ad000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0017070 con 0x7f7ae806d5f0 2024-03-31T15:54:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad000ca00 con 0x7f7ae806d5f0 2024-03-31T15:54:39.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad000cdd0 con 0x7f7ae806d5f0 2024-03-31T15:54:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae815d320 con 0x7f7ae806d5f0 2024-03-31T15:54:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae80c8370 con 0x7f7ae806d5f0 2024-03-31T15:54:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae80686d0 con 0x7f7ae806d5f0 2024-03-31T15:54:39.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.617+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7ad001a070 con 0x7f7ae806d5f0 2024-03-31T15:54:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.621+0000 7f7adfe00640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 0x7f7ac406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.621+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7ad0086ed0 con 0x7f7ae806d5f0 2024-03-31T15:54:39.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.621+0000 7f7ae7400640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 0x7f7ac406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.621+0000 7f7ae7400640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 0x7f7ac406a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7ae81fbac0 tx=0x7f7ad8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.625+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7ad0056a30 con 0x7f7ae806d5f0 2024-03-31T15:54:39.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.781+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ae80b1b60 con 0x7f7ae806d5f0 2024-03-31T15:54:39.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.781+0000 7f7adfe00640 1 -- 172.21.15.90:0/1742246170 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f7ad005ad30 con 0x7f7ae806d5f0 2024-03-31T15:54:39.830 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:39.830 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:39.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 msgr2=0x7f7ac406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 0x7f7ac406a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7ae81fbac0 tx=0x7f7ad8009040 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 msgr2=0x7f7ae815c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000e930 tx=0x7f7ad000edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 shutdown_connections 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7ac40683b0 0x7f7ac406a870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7ae815e1b0 0x7f7ae80b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7ae806d5f0 0x7f7ae815c680 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 --2- 172.21.15.90:0/1742246170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ae806c7b0 0x7f7ae80c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 >> 172.21.15.90:0/1742246170 conn(0x7f7ae809aa40 msgr2=0x7f7ae815f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 shutdown_connections 2024-03-31T15:54:39.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.785+0000 7f7aee200640 1 -- 172.21.15.90:0/1742246170 wait complete. 2024-03-31T15:54:39.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:39.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 Processor -- start 2024-03-31T15:54:39.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 -- start start 2024-03-31T15:54:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 0x7fbeb015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 0x7fbeb00b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb0070910 con 0x7fbeb015f5c0 2024-03-31T15:54:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbeb00b37e0 con 0x7fbeb015aef0 2024-03-31T15:54:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeb4a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb0070bf0 con 0x7fbeb00b2e30 2024-03-31T15:54:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbea5400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbeb015f5c0 0x7fbeb015c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38272/0 (socket says 172.21.15.90:38272) 2024-03-31T15:54:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbea5400640 1 -- 172.21.15.90:0/4087693179 learned_addr learned my addr 172.21.15.90:0/4087693179 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeae800640 1 --2- 172.21.15.90:0/4087693179 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 0x7fbeb00b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeade00640 1 --2- 172.21.15.90:0/4087693179 >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 0x7fbeb015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbead400640 1 -- 172.21.15.90:0/4087693179 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244868576 0 0) 0x7fbeb0070910 con 0x7fbeb015f5c0 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbead400640 1 -- 172.21.15.90:0/4087693179 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe94003530 con 0x7fbeb015f5c0 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.981+0000 7fbeae800640 1 -- 172.21.15.90:0/4087693179 >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 msgr2=0x7fbeb015b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeae800640 1 --2- 172.21.15.90:0/4087693179 >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 0x7fbeb015b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeae800640 1 -- 172.21.15.90:0/4087693179 >> v1:172.21.15.90:6789/0 conn(0x7fbeb015f5c0 legacy=0x7fbeb015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeae800640 1 -- 172.21.15.90:0/4087693179 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb015b9d0 con 0x7fbeb00b2e30 2024-03-31T15:54:39.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeade00640 1 --2- 172.21.15.90:0/4087693179 >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 0x7fbeb015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeae800640 1 --2- 172.21.15.90:0/4087693179 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 0x7fbeb00b32a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009dc0 tx=0x7fbea801c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35e1fb83b707faba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbead400640 1 -- 172.21.15.90:0/4087693179 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea80b2070 con 0x7fbeb00b2e30 2024-03-31T15:54:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbead400640 1 -- 172.21.15.90:0/4087693179 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea80a5920 con 0x7fbeb00b2e30 2024-03-31T15:54:39.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbead400640 1 -- 172.21.15.90:0/4087693179 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea80a5cf0 con 0x7fbeb00b2e30 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeb4a00640 1 -- 172.21.15.90:0/4087693179 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 msgr2=0x7fbeb00b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.985+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/4087693179 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 0x7fbeb00b32a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbea8009dc0 tx=0x7fbea801c950 comp rx=0 tx=0).stop 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- 172.21.15.90:0/4087693179 shutdown_connections 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/4087693179 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb00b2e30 0x7fbeb00b32a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/4087693179 >> v2:172.21.15.102:3300/0 conn(0x7fbeb015aef0 0x7fbeb015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- 172.21.15.90:0/4087693179 >> 172.21.15.90:0/4087693179 conn(0x7fbeb00cde90 msgr2=0x7fbeb00ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:39.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- 172.21.15.90:0/4087693179 shutdown_connections 2024-03-31T15:54:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- 172.21.15.90:0/4087693179 wait complete. 2024-03-31T15:54:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 Processor -- start 2024-03-31T15:54:39.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- start start 2024-03-31T15:54:39.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 0x7fbeb0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb015f5c0 0x7fbeb0154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb00d0a10 con 0x7fbeb00b2e30 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb00d0890 con 0x7fbeb015aef0 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb00af430 con 0x7fbeb015f5c0 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43264/0 (socket says 172.21.15.90:43264) 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 -- 172.21.15.90:0/2617041255 learned_addr learned my addr 172.21.15.90:0/2617041255 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeae800640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb015f5c0 0x7fbeb0154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbea5400640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 0x7fbeb0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 -- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb015f5c0 msgr2=0x7fbeb0154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb015f5c0 0x7fbeb0154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 -- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 msgr2=0x7fbeb0150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 0x7fbeb0150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 -- 172.21.15.90:0/2617041255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb0164680 con 0x7fbeb00b2e30 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbea5400640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 0x7fbeb0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeade00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbea000ebe0 tx=0x7fbea000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0017070 con 0x7fbeb00b2e30 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea000cc30 con 0x7fbeb00b2e30 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.989+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb0164960 con 0x7fbeb00b2e30 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0013470 con 0x7fbeb00b2e30 2024-03-31T15:54:39.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb0164f20 con 0x7fbeb00b2e30 2024-03-31T15:54:39.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb00685f0 con 0x7fbeb00b2e30 2024-03-31T15:54:39.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbea001a070 con 0x7fbeb00b2e30 2024-03-31T15:54:39.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbea6a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 0x7fbe8c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:39.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbea0086d10 con 0x7fbeb00b2e30 2024-03-31T15:54:39.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.993+0000 7fbea5400640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 0x7fbe8c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:39.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.997+0000 7fbea5400640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 0x7fbe8c06a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbeb015b410 tx=0x7fbe98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:39.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:39.997+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbea008c020 con 0x7fbeb00b2e30 2024-03-31T15:54:40.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.157+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbeb00cf760 con 0x7fbeb00b2e30 2024-03-31T15:54:40.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbea6a00640 1 -- 172.21.15.90:0/2617041255 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbea00567e0 con 0x7fbeb00b2e30 2024-03-31T15:54:40.152 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 msgr2=0x7fbe8c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 0x7fbe8c06a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbeb015b410 tx=0x7fbe98009040 comp rx=0 tx=0).stop 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 msgr2=0x7fbeb0163ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbea000ebe0 tx=0x7fbea000c520 comp rx=0 tx=0).stop 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 shutdown_connections 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbe8c0683b0 0x7fbe8c06a870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbeb015f5c0 0x7fbeb0154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbeb015aef0 0x7fbeb0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 --2- 172.21.15.90:0/2617041255 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbeb00b2e30 0x7fbeb0163ff0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 >> 172.21.15.90:0/2617041255 conn(0x7fbeb00cde90 msgr2=0x7fbeb00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 shutdown_connections 2024-03-31T15:54:40.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.161+0000 7fbeb4a00640 1 -- 172.21.15.90:0/2617041255 wait complete. 2024-03-31T15:54:40.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 Processor -- start 2024-03-31T15:54:40.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 -- start start 2024-03-31T15:54:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6b1006d020 0x7f6b100ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 0x7f6b100cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b1015e8c0 con 0x7f6b1015e4e0 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6b100d0ff0 con 0x7f6b1006d020 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b15c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6b100d12d0 con 0x7f6b1006d490 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b0ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6b1015e4e0 0x7f6b1006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38280/0 (socket says 172.21.15.90:38280) 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b0ea00640 1 -- 172.21.15.90:0/210524791 learned_addr learned my addr 172.21.15.90:0/210524791 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.321+0000 7f6b0e000640 1 --2- 172.21.15.90:0/210524791 >> v2:172.21.15.90:3300/0 conn(0x7f6b1006d020 0x7f6b100ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0f400640 1 --2- 172.21.15.90:0/210524791 >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 0x7f6b100cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0e000640 1 -- 172.21.15.90:0/210524791 >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 msgr2=0x7f6b100cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0e000640 1 --2- 172.21.15.90:0/210524791 >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 0x7f6b100cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0e000640 1 -- 172.21.15.90:0/210524791 >> v1:172.21.15.90:6789/0 conn(0x7f6b1015e4e0 legacy=0x7f6b1006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0e000640 1 -- 172.21.15.90:0/210524791 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b100cc710 con 0x7f6b1006d020 2024-03-31T15:54:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0d600640 1 -- 172.21.15.90:0/210524791 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957963001 0 0) 0x7f6b1015e8c0 con 0x7f6b1015e4e0 2024-03-31T15:54:40.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0f400640 1 --2- 172.21.15.90:0/210524791 >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 0x7f6b100cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:40.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0e000640 1 --2- 172.21.15.90:0/210524791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1006d020 0x7f6b100ccf10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0097c0 tx=0x7f6afc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c156b34440cef476 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0d600640 1 -- 172.21.15.90:0/210524791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0b0070 con 0x7f6b1006d020 2024-03-31T15:54:40.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0d600640 1 -- 172.21.15.90:0/210524791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6afc0ab600 con 0x7f6b1006d020 2024-03-31T15:54:40.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.325+0000 7f6b0d600640 1 -- 172.21.15.90:0/210524791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0a4e60 con 0x7f6b1006d020 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- 172.21.15.90:0/210524791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1006d020 msgr2=0x7f6b100ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- 172.21.15.90:0/210524791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1006d020 0x7f6b100ccf10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0097c0 tx=0x7f6afc01c3d0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- 172.21.15.90:0/210524791 shutdown_connections 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- 172.21.15.90:0/210524791 >> v2:172.21.15.102:3300/0 conn(0x7f6b1006d490 0x7f6b100cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- 172.21.15.90:0/210524791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1006d020 0x7f6b100ccf10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- 172.21.15.90:0/210524791 >> 172.21.15.90:0/210524791 conn(0x7f6b100d1f20 msgr2=0x7f6b100d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- 172.21.15.90:0/210524791 shutdown_connections 2024-03-31T15:54:40.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- 172.21.15.90:0/210524791 wait complete. 2024-03-31T15:54:40.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 Processor -- start 2024-03-31T15:54:40.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- start start 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b1006d020 0x7f6b10156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b1006d490 0x7f6b100cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b100d0d90 con 0x7f6b1015e4e0 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6b100d0c10 con 0x7f6b1006d020 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b15c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b1016b030 con 0x7f6b1006d490 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43278/0 (socket says 172.21.15.90:43278) 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 -- 172.21.15.90:0/1526276423 learned_addr learned my addr 172.21.15.90:0/1526276423 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0ea00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b1006d020 0x7f6b10156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0e000640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b1006d490 0x7f6b100cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 -- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b1006d490 msgr2=0x7f6b100cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b1006d490 0x7f6b100cb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 -- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b1006d020 msgr2=0x7f6b10156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b1006d020 0x7f6b10156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 -- 172.21.15.90:0/1526276423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b10153ed0 con 0x7f6b1015e4e0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.329+0000 7f6b0f400640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800cf20 tx=0x7f6b0800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0801a070 con 0x7f6b1015e4e0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b0800d840 con 0x7f6b1015e4e0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b10155740 con 0x7f6b1015e4e0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0800dc30 con 0x7f6b1015e4e0 2024-03-31T15:54:40.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b10155d00 con 0x7f6b1015e4e0 2024-03-31T15:54:40.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b100685f0 con 0x7f6b1015e4e0 2024-03-31T15:54:40.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b0801d070 con 0x7f6b1015e4e0 2024-03-31T15:54:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 0x7f6af006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6b080876f0 con 0x7f6b1015e4e0 2024-03-31T15:54:40.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.333+0000 7f6b0ea00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 0x7f6af006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.337+0000 7f6b0ea00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 0x7f6af006a840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b00000f70 tx=0x7f6b00007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.337+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6b08057810 con 0x7f6b1015e4e0 2024-03-31T15:54:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.497+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b1015b370 con 0x7f6b1015e4e0 2024-03-31T15:54:40.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.497+0000 7f6b06a00640 1 -- 172.21.15.90:0/1526276423 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f6b08057110 con 0x7f6b1015e4e0 2024-03-31T15:54:40.491 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 msgr2=0x7f6af006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 0x7f6af006a840 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6b00000f70 tx=0x7f6b00007b80 comp rx=0 tx=0).stop 2024-03-31T15:54:40.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 msgr2=0x7f6b100c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800cf20 tx=0x7f6b0800a7d0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 shutdown_connections 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6af0068380 0x7f6af006a840 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b1015e4e0 0x7f6b100c9680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b1006d490 0x7f6b100cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 --2- 172.21.15.90:0/1526276423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b1006d020 0x7f6b10156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 >> 172.21.15.90:0/1526276423 conn(0x7f6b100d1f20 msgr2=0x7f6b1015a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 shutdown_connections 2024-03-31T15:54:40.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.501+0000 7f6b15c00640 1 -- 172.21.15.90:0/1526276423 wait complete. 2024-03-31T15:54:40.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:40.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 Processor -- start 2024-03-31T15:54:40.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 -- start start 2024-03-31T15:54:40.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5d815c320 0x7fa5d80cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80cefc0 con 0x7fa5d80cd640 2024-03-31T15:54:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d80cf2a0 con 0x7fa5d80d2b20 2024-03-31T15:54:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5e0000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d80aed70 con 0x7fa5d815c320 2024-03-31T15:54:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5de000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa5d80cd640 0x7fa5d80af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38284/0 (socket says 172.21.15.90:38284) 2024-03-31T15:54:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5de000640 1 -- 172.21.15.90:0/2921225877 learned_addr learned my addr 172.21.15.90:0/2921225877 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:40.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5dd600640 1 --2- 172.21.15.90:0/2921225877 >> v2:172.21.15.102:3300/0 conn(0x7fa5d815c320 0x7fa5d80cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931595349 0 0) 0x7fa5d80cefc0 con 0x7fa5d80cd640 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5c0003530 con 0x7fa5d80cd640 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1307770592 0 0) 0x7fa5c0003530 con 0x7fa5d80cd640 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 >> v2:172.21.15.102:3300/0 conn(0x7fa5d815c320 msgr2=0x7fa5d80cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 --2- 172.21.15.90:0/2921225877 >> v2:172.21.15.102:3300/0 conn(0x7fa5d815c320 0x7fa5d80cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 >> v1:172.21.15.102:6789/0 conn(0x7fa5d80d2b20 legacy=0x7fa5d80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d80cf580 con 0x7fa5d80cd640 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fa5d0001a00 con 0x7fa5d80cd640 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 >> v1:172.21.15.90:6789/0 conn(0x7fa5d80cd640 legacy=0x7fa5d80af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c00039c0 0x7fa5c0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5c00043c0 0x7fa5c0004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 0x7fa5c000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d80aed70 con 0x7fa5c0005130 2024-03-31T15:54:40.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c000c070 con 0x7fa5c00039c0 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.657+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c000c270 con 0x7fa5c00043c0 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 0x7fa5c000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5dd600640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c00039c0 0x7fa5c0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5dcc00640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5c00043c0 0x7fa5c0004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 -- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5c00043c0 msgr2=0x7fa5c0004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5c00043c0 0x7fa5c0004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 -- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c00039c0 msgr2=0x7fa5c0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c00039c0 0x7fa5c0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 -- 172.21.15.90:0/2921225877 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c0003530 con 0x7fa5c0005130 2024-03-31T15:54:40.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5dd600640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c00039c0 0x7fa5c0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 -- 172.21.15.90:0/2921225877 reap_dead start 2024-03-31T15:54:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5de000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 0x7fa5c000b9d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000a7e0 tx=0x7fa5d001d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5202b5fb1811eac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0096b0 con 0x7fa5c0005130 2024-03-31T15:54:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d80d0f20 con 0x7fa5c0005130 2024-03-31T15:54:40.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.661+0000 7fa5d7e00640 1 -- 172.21.15.90:0/2921225877 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000a430 con 0x7fa5c0005130 2024-03-31T15:54:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 msgr2=0x7fa5c000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 0x7fa5c000b9d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000a7e0 tx=0x7fa5d001d2e0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- 172.21.15.90:0/2921225877 shutdown_connections 2024-03-31T15:54:40.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 --2- 172.21.15.90:0/2921225877 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5c0005130 0x7fa5c000b9d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- 172.21.15.90:0/2921225877 >> 172.21.15.90:0/2921225877 conn(0x7fa5d80d1f20 msgr2=0x7fa5d80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- 172.21.15.90:0/2921225877 shutdown_connections 2024-03-31T15:54:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- 172.21.15.90:0/2921225877 wait complete. 2024-03-31T15:54:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 Processor -- start 2024-03-31T15:54:40.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- start start 2024-03-31T15:54:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c0005130 0x7fa5d80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5d80b2380 0x7fa5d80b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d80b04b0 con 0x7fa5d80b1380 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d80b0330 con 0x7fa5c0005130 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5e0000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d80af1e0 con 0x7fa5d80b2380 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43292/0 (socket says 172.21.15.90:43292) 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 -- 172.21.15.90:0/3186528282 learned_addr learned my addr 172.21.15.90:0/3186528282 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5dcc00640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5d80b2380 0x7fa5d80b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5dd600640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c0005130 0x7fa5d80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 -- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5d80b2380 msgr2=0x7fa5d80b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5d80b2380 0x7fa5d80b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 -- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c0005130 msgr2=0x7fa5d80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c0005130 0x7fa5d80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.665+0000 7fa5de000640 1 -- 172.21.15.90:0/3186528282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d8153e40 con 0x7fa5d80b1380 2024-03-31T15:54:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5de000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa5d001db80 tx=0x7fa5c0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000a430 con 0x7fa5d80b1380 2024-03-31T15:54:40.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d00b4070 con 0x7fa5d80b1380 2024-03-31T15:54:40.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d8154120 con 0x7fa5d80b1380 2024-03-31T15:54:40.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00ad670 con 0x7fa5d80b1380 2024-03-31T15:54:40.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d81546e0 con 0x7fa5d80b1380 2024-03-31T15:54:40.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d80685f0 con 0x7fa5d80b1380 2024-03-31T15:54:40.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.669+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa5d0007070 con 0x7fa5d80b1380 2024-03-31T15:54:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.673+0000 7fa5d5600640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 0x7fa5b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.673+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa5d0120410 con 0x7fa5d80b1380 2024-03-31T15:54:40.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.673+0000 7fa5dd600640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 0x7fa5b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.673+0000 7fa5dd600640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 0x7fa5b806a840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8070c20 tx=0x7fa5cc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.677+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa5d0016ff0 con 0x7fa5d80b1380 2024-03-31T15:54:40.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa5d815f780 con 0x7fa5d80b1380 2024-03-31T15:54:40.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5d5600640 1 -- 172.21.15.90:0/3186528282 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fa5d00019c0 con 0x7fa5d80b1380 2024-03-31T15:54:40.826 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 msgr2=0x7fa5b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 0x7fa5b806a840 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8070c20 tx=0x7fa5cc009190 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 msgr2=0x7fa5d80b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa5d001db80 tx=0x7fa5c0005840 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 shutdown_connections 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa5b8068380 0x7fa5b806a840 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa5d80b1380 0x7fa5d80b1830 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa5d80b2380 0x7fa5d80b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 --2- 172.21.15.90:0/3186528282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5c0005130 0x7fa5d80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 >> 172.21.15.90:0/3186528282 conn(0x7fa5d80d1f20 msgr2=0x7fa5d80cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 shutdown_connections 2024-03-31T15:54:40.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.833+0000 7fa5e0000640 1 -- 172.21.15.90:0/3186528282 wait complete. 2024-03-31T15:54:40.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 Processor -- start 2024-03-31T15:54:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 -- start start 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f252c15e4e0 0x7f252c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c06ce50 0x7f252c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252c0ce7a0 0x7f252c0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f252c15e8c0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f252c0d0ff0 con 0x7f252c15e4e0 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2533200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f252c0d12d0 con 0x7f252c06ce50 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2531200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252c0ce7a0 0x7f252c0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2531200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252c0ce7a0 0x7f252c0cec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43296/0 (socket says 172.21.15.90:43296) 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2531200640 1 -- 172.21.15.90:0/70324485 learned_addr learned my addr 172.21.15.90:0/70324485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:40.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f2523e00640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c06ce50 0x7f252c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.993+0000 7f252be00640 1 --2- 172.21.15.90:0/70324485 >> v2:172.21.15.102:3300/0 conn(0x7f252c15e4e0 0x7f252c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 -- 172.21.15.90:0/70324485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c06ce50 msgr2=0x7f252c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c06ce50 0x7f252c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 -- 172.21.15.90:0/70324485 >> v2:172.21.15.102:3300/0 conn(0x7f252c15e4e0 msgr2=0x7f252c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 --2- 172.21.15.90:0/70324485 >> v2:172.21.15.102:3300/0 conn(0x7f252c15e4e0 0x7f252c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 -- 172.21.15.90:0/70324485 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c0cc710 con 0x7f252c0ce7a0 2024-03-31T15:54:40.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2531200640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c0cec30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f252c0d3020 tx=0x7f252401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750226966c5feb45 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f252b400640 1 -- 172.21.15.90:0/70324485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25240a55e0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f252b400640 1 -- 172.21.15.90:0/70324485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25240a5780 con 0x7f252c0ce7a0 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f252b400640 1 -- 172.21.15.90:0/70324485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25240a5b50 con 0x7f252c0ce7a0 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 -- 172.21.15.90:0/70324485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 msgr2=0x7f252c0cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c0cec30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f252c0d3020 tx=0x7f252401c960 comp rx=0 tx=0).stop 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 -- 172.21.15.90:0/70324485 shutdown_connections 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c0cec30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 --2- 172.21.15.90:0/70324485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c06ce50 0x7f252c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 --2- 172.21.15.90:0/70324485 >> v2:172.21.15.102:3300/0 conn(0x7f252c15e4e0 0x7f252c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 -- 172.21.15.90:0/70324485 >> 172.21.15.90:0/70324485 conn(0x7f252c0d1f20 msgr2=0x7f252c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 -- 172.21.15.90:0/70324485 shutdown_connections 2024-03-31T15:54:40.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:40.997+0000 7f2533200640 1 -- 172.21.15.90:0/70324485 wait complete. 2024-03-31T15:54:40.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 Processor -- start 2024-03-31T15:54:40.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- start start 2024-03-31T15:54:40.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 0x7f252c09dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c15e4e0 0x7f252c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f252c0cd090 con 0x7f252c0ce7a0 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f252c0ccf10 con 0x7f252c06ce50 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f252c0cd210 con 0x7f252c15e4e0 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43306/0 (socket says 172.21.15.90:43306) 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 -- 172.21.15.90:0/695259851 learned_addr learned my addr 172.21.15.90:0/695259851 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2531200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c15e4e0 0x7f252c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f252be00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 0x7f252c09dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 -- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c15e4e0 msgr2=0x7f252c06f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c15e4e0 0x7f252c06f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 -- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 msgr2=0x7f252c09dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:40.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 0x7f252c09dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 -- 172.21.15.90:0/695259851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c09e410 con 0x7f252c0ce7a0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f252be00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 0x7f252c09dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2523e00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f251400e930 tx=0x7f251400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514017070 con 0x7f252c0ce7a0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251400ca00 con 0x7f252c0ce7a0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252c09e6f0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251400cdb0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.001+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252c09ecb0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.005+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252c0685f0 con 0x7f252c0ce7a0 2024-03-31T15:54:40.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.005+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f251401a070 con 0x7f252c0ce7a0 2024-03-31T15:54:40.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.005+0000 7f2528c00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 0x7f250806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:40.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.005+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2514086e20 con 0x7f252c0ce7a0 2024-03-31T15:54:40.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.005+0000 7f252be00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 0x7f250806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:40.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.009+0000 7f252be00640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 0x7f250806a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f251c0099c0 tx=0x7f251c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.009+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f25140570a0 con 0x7f252c0ce7a0 2024-03-31T15:54:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f252c1578b0 con 0x7f252c0ce7a0 2024-03-31T15:54:41.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2528c00640 1 -- 172.21.15.90:0/695259851 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f25140569a0 con 0x7f252c0ce7a0 2024-03-31T15:54:41.162 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 msgr2=0x7f250806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 0x7f250806a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f251c0099c0 tx=0x7f251c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 msgr2=0x7f252c09f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f251400e930 tx=0x7f251400edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 shutdown_connections 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2508068380 0x7f250806a840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252c15e4e0 0x7f252c06f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252c0ce7a0 0x7f252c09f4b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 --2- 172.21.15.90:0/695259851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252c06ce50 0x7f252c09dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 >> 172.21.15.90:0/695259851 conn(0x7f252c0d1f20 msgr2=0x7f252c0c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 shutdown_connections 2024-03-31T15:54:41.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.169+0000 7f2533200640 1 -- 172.21.15.90:0/695259851 wait complete. 2024-03-31T15:54:41.190 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-31T15:54:41.191 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:54:41.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.333+0000 7f5683a00640 1 Processor -- start 2024-03-31T15:54:41.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.333+0000 7f5683a00640 1 -- start start 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5683a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f567c15c120 0x7f567c15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5683a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f567c0d2680 0x7f567c0b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5683a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567c0d2b10 con 0x7f567c0b2c20 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5683a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f567c160bf0 con 0x7f567c0d2680 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5683a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f567c160350 con 0x7f567c15c120 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f567c0d2680 0x7f567c0b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f567c0d2680 0x7f567c0b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43320/0 (socket says 172.21.15.90:43320) 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 -- 172.21.15.90:0/4195793413 learned_addr learned my addr 172.21.15.90:0/4195793413 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:41.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f5681000640 1 --2- 172.21.15.90:0/4195793413 >> v2:172.21.15.102:3300/0 conn(0x7f567c15c120 0x7f567c15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 -- 172.21.15.90:0/4195793413 >> v2:172.21.15.102:3300/0 conn(0x7f567c15c120 msgr2=0x7f567c15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 --2- 172.21.15.90:0/4195793413 >> v2:172.21.15.102:3300/0 conn(0x7f567c15c120 0x7f567c15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 -- 172.21.15.90:0/4195793413 >> v1:172.21.15.102:6789/0 conn(0x7f567c0b2c20 legacy=0x7f567c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:41.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.337+0000 7f567be00640 1 -- 172.21.15.90:0/4195793413 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c160630 con 0x7f567c0d2680 2024-03-31T15:54:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f567be00640 1 --2- 172.21.15.90:0/4195793413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0d2680 0x7f567c0b26e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f567c0d2d20 tx=0x7f566801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9433000ef07e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f567b400640 1 -- 172.21.15.90:0/4195793413 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680ab620 con 0x7f567c0d2680 2024-03-31T15:54:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f567b400640 1 -- 172.21.15.90:0/4195793413 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56680b0070 con 0x7f567c0d2680 2024-03-31T15:54:41.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f567b400640 1 -- 172.21.15.90:0/4195793413 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680ab9e0 con 0x7f567c0d2680 2024-03-31T15:54:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- 172.21.15.90:0/4195793413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0d2680 msgr2=0x7f567c0b26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 --2- 172.21.15.90:0/4195793413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0d2680 0x7f567c0b26e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f567c0d2d20 tx=0x7f566801c420 comp rx=0 tx=0).stop 2024-03-31T15:54:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- 172.21.15.90:0/4195793413 shutdown_connections 2024-03-31T15:54:41.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 --2- 172.21.15.90:0/4195793413 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0d2680 0x7f567c0b26e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 --2- 172.21.15.90:0/4195793413 >> v2:172.21.15.102:3300/0 conn(0x7f567c15c120 0x7f567c15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- 172.21.15.90:0/4195793413 >> 172.21.15.90:0/4195793413 conn(0x7f567c09aa40 msgr2=0x7f567c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- 172.21.15.90:0/4195793413 shutdown_connections 2024-03-31T15:54:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- 172.21.15.90:0/4195793413 wait complete. 2024-03-31T15:54:41.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 Processor -- start 2024-03-31T15:54:41.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 -- start start 2024-03-31T15:54:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.341+0000 7f5683a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567c0d2680 0x7f567c151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f567c15c120 0x7f567c15a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f567c1691c0 con 0x7f567c0b2c20 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f567c169040 con 0x7f567c0d2680 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f567c169340 con 0x7f567c15c120 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43332/0 (socket says 172.21.15.90:43332) 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 -- 172.21.15.90:0/868497931 learned_addr learned my addr 172.21.15.90:0/868497931 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f567be00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567c0d2680 0x7f567c151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f567c15c120 0x7f567c15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 -- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f567c15c120 msgr2=0x7f567c15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f567c15c120 0x7f567c15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 -- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567c0d2680 msgr2=0x7f567c151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567c0d2680 0x7f567c151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 -- 172.21.15.90:0/868497931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567c152820 con 0x7f567c0b2c20 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5681000640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f567000d5f0 tx=0x7f567000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56700125f0 con 0x7f567c0b2c20 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5670019070 con 0x7f567c0b2c20 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567c0cb680 con 0x7f567c0b2c20 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56700129b0 con 0x7f567c0b2c20 2024-03-31T15:54:41.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567c0cbc40 con 0x7f567c0b2c20 2024-03-31T15:54:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567c0cbe40 con 0x7f567c0b2c20 2024-03-31T15:54:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5670012b50 con 0x7f567c0b2c20 2024-03-31T15:54:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.345+0000 7f5678c00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 0x7f565c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.349+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5670087120 con 0x7f567c0b2c20 2024-03-31T15:54:41.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.349+0000 7f567be00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 0x7f565c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.349+0000 7f567be00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 0x7f565c06a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56680af000 tx=0x7f56680a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.353+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f567c0cbe40 con 0x7f567c0b2c20 2024-03-31T15:54:41.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.509+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f567c0686d0 con 0x7f567c0b2c20 2024-03-31T15:54:41.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.509+0000 7f5678c00640 1 -- 172.21.15.90:0/868497931 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f56700572e0 con 0x7f567c0b2c20 2024-03-31T15:54:41.503 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:41.504 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 msgr2=0x7f565c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 0x7f565c06a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f56680af000 tx=0x7f56680a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 msgr2=0x7f567c1515c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f567000d5f0 tx=0x7f567000dab0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 shutdown_connections 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c068380 0x7f565c06a840 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f567c15c120 0x7f567c15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567c0d2680 0x7f567c151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 --2- 172.21.15.90:0/868497931 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f567c0b2c20 0x7f567c1515c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 >> 172.21.15.90:0/868497931 conn(0x7f567c09aa40 msgr2=0x7f567c070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 shutdown_connections 2024-03-31T15:54:41.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.513+0000 7f5683a00640 1 -- 172.21.15.90:0/868497931 wait complete. 2024-03-31T15:54:41.532 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:41.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:41.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.677+0000 7f3556600640 1 Processor -- start 2024-03-31T15:54:41.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.677+0000 7f3556600640 1 -- start start 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3556600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f35500cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3556600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500cefc0 con 0x7f35500cd640 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3556600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500cf2a0 con 0x7f355015c320 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3556600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f35500aed70 con 0x7f35500d2b20 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f35500cd640 0x7f35500af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38288/0 (socket says 172.21.15.90:38288) 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 -- 172.21.15.90:0/1048122213 learned_addr learned my addr 172.21.15.90:0/1048122213 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3547e00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f35500cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253293398 0 0) 0x7f35500cefc0 con 0x7f35500cd640 2024-03-31T15:54:41.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3534003530 con 0x7f35500cd640 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385648938 0 0) 0x7f35500cf2a0 con 0x7f355015c320 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35500cefc0 con 0x7f355015c320 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2099465155 0 0) 0x7f3534003530 con 0x7f35500cd640 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 msgr2=0x7f35500cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f35500cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 >> v1:172.21.15.102:6789/0 conn(0x7f355015c320 legacy=0x7f35500cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35500cf580 con 0x7f35500cd640 2024-03-31T15:54:41.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f3548001a00 con 0x7f35500cd640 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 >> v1:172.21.15.90:6789/0 conn(0x7f35500cd640 legacy=0x7f35500af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f3534003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35340040e0 0x7f3534004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3534004d50 0x7f35340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35500aed70 con 0x7f3534004d50 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f353400bb60 con 0x7f35340040e0 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f353400bd30 con 0x7f35500d2b20 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3534004d50 0x7f35340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f3547e00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35340040e0 0x7f3534004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354f400640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f3534003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 -- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 msgr2=0x7f3534003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f35500d2b20 0x7f3534003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 -- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35340040e0 msgr2=0x7f3534004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35340040e0 0x7f3534004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.681+0000 7f354fe00640 1 -- 172.21.15.90:0/1048122213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3534003530 con 0x7f3534004d50 2024-03-31T15:54:41.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.685+0000 7f354fe00640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3534004d50 0x7f35340051d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f354800a7d0 tx=0x7f354801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d1e99465364179 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.685+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354800a430 con 0x7f3534004d50 2024-03-31T15:54:41.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.685+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35480ad630 con 0x7f3534004d50 2024-03-31T15:54:41.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.685+0000 7f354ea00640 1 -- 172.21.15.90:0/1048122213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35480ada60 con 0x7f3534004d50 2024-03-31T15:54:41.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- 172.21.15.90:0/1048122213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3534004d50 msgr2=0x7f35340051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 --2- 172.21.15.90:0/1048122213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3534004d50 0x7f35340051d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f354800a7d0 tx=0x7f354801d2e0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f354fe00640 1 -- 172.21.15.90:0/1048122213 reap_dead start 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- 172.21.15.90:0/1048122213 shutdown_connections 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- 172.21.15.90:0/1048122213 >> 172.21.15.90:0/1048122213 conn(0x7f35500d1f20 msgr2=0x7f35500d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- 172.21.15.90:0/1048122213 shutdown_connections 2024-03-31T15:54:41.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- 172.21.15.90:0/1048122213 wait complete. 2024-03-31T15:54:41.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 Processor -- start 2024-03-31T15:54:41.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- start start 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3534004d50 0x7f35500cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35501f7c60 0x7f35500c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f35500d0e90 con 0x7f35500c84f0 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f35500d0d10 con 0x7f35501f7c60 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3556600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f35500cd100 con 0x7f3534004d50 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43346/0 (socket says 172.21.15.90:43346) 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 -- 172.21.15.90:0/1771254654 learned_addr learned my addr 172.21.15.90:0/1771254654 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f354f400640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3534004d50 0x7f35500cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f354fe00640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35501f7c60 0x7f35500c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 -- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3534004d50 msgr2=0x7f35500cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3534004d50 0x7f35500cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 -- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35501f7c60 msgr2=0x7f35500c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.689+0000 7f3547e00640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35501f7c60 0x7f35500c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547e00640 1 -- 172.21.15.90:0/1771254654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35500ca6d0 con 0x7f35500c84f0 2024-03-31T15:54:41.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547e00640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3538011a00 tx=0x7f3538011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353800fb50 con 0x7f35500c84f0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f353800fcf0 con 0x7f35500c84f0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353800fea0 con 0x7f35500c84f0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35500ca9b0 con 0x7f35500c84f0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35500cb9d0 con 0x7f35500c84f0 2024-03-31T15:54:41.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35500685f0 con 0x7f35500c84f0 2024-03-31T15:54:41.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.693+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3538012040 con 0x7f35500c84f0 2024-03-31T15:54:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.697+0000 7f3547400640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 0x7f352c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:41.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.697+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3538087290 con 0x7f35500c84f0 2024-03-31T15:54:41.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.697+0000 7f354f400640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 0x7f352c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:41.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.697+0000 7f354f400640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 0x7f352c06a870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3540005630 tx=0x7f35400014b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:41.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.701+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3538019030 con 0x7f35500c84f0 2024-03-31T15:54:41.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.857+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35501f8040 con 0x7f35500c84f0 2024-03-31T15:54:41.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.857+0000 7f3547400640 1 -- 172.21.15.90:0/1771254654 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3538056ec0 con 0x7f35500c84f0 2024-03-31T15:54:41.851 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 msgr2=0x7f352c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 0x7f352c06a870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3540005630 tx=0x7f35400014b0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 msgr2=0x7f35501f7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3538011a00 tx=0x7f3538011ec0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 shutdown_connections 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f352c0683b0 0x7f352c06a870 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35501f7c60 0x7f35500c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35500c84f0 0x7f35501f7720 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 --2- 172.21.15.90:0/1771254654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3534004d50 0x7f35500cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:41.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 >> 172.21.15.90:0/1771254654 conn(0x7f35500d1f20 msgr2=0x7f35500af100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:41.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 shutdown_connections 2024-03-31T15:54:41.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:41.861+0000 7f3556600640 1 -- 172.21.15.90:0/1771254654 wait complete. 2024-03-31T15:54:41.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:42.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 Processor -- start 2024-03-31T15:54:42.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 -- start start 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1dc00d2b20 0x7f1dc00cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1dc00af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc00cefc0 con 0x7f1dc015c320 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc00aed70 con 0x7f1dc00d2b20 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dc5800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc00cf2a0 con 0x7f1dc00cd640 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1dc015c320 0x7f1dc00cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38290/0 (socket says 172.21.15.90:38290) 2024-03-31T15:54:42.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbea00640 1 -- 172.21.15.90:0/4100857336 learned_addr learned my addr 172.21.15.90:0/4100857336 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1dc00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1db7e00640 1 --2- 172.21.15.90:0/4100857336 >> v2:172.21.15.102:3300/0 conn(0x7f1dc00d2b20 0x7f1dc00cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289683638 0 0) 0x7f1dc00cefc0 con 0x7f1dc015c320 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1da4003530 con 0x7f1dc015c320 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3967270401 0 0) 0x7f1da4003530 con 0x7f1dc015c320 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 msgr2=0x7f1dc00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1dc00af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 >> v2:172.21.15.102:3300/0 conn(0x7f1dc00d2b20 msgr2=0x7f1dc00cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 --2- 172.21.15.90:0/4100857336 >> v2:172.21.15.102:3300/0 conn(0x7f1dc00d2b20 0x7f1dc00cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc00cf580 con 0x7f1dc015c320 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f1db0001680 con 0x7f1dc015c320 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 >> v1:172.21.15.90:6789/0 conn(0x7f1dc015c320 legacy=0x7f1dc00cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1da40039c0 0x7f1da4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1da4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da4004d50 0x7f1da40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00aed70 con 0x7f1da4004d50 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00cf2a0 con 0x7f1da40039c0 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1da400ba00 con 0x7f1dc00cd640 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da4004d50 0x7f1da40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbea00640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1da40039c0 0x7f1da4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1db7e00640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1da4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 -- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 msgr2=0x7f1da4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1da4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 -- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1da40039c0 msgr2=0x7f1da4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1da40039c0 0x7f1da4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1dbf400640 1 -- 172.21.15.90:0/4100857336 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da4003530 con 0x7f1da4004d50 2024-03-31T15:54:42.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.021+0000 7f1db7e00640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1dc00cd640 0x7f1da4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:42.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dbf400640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da4004d50 0x7f1da40051d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009f90 tx=0x7f1db801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdcfc89eb1d58e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b0070 con 0x7f1da4004d50 2024-03-31T15:54:42.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc00d0f20 con 0x7f1da4004d50 2024-03-31T15:54:42.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dbe000640 1 -- 172.21.15.90:0/4100857336 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8009c10 con 0x7f1da4004d50 2024-03-31T15:54:42.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 -- 172.21.15.90:0/4100857336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da4004d50 msgr2=0x7f1da40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 --2- 172.21.15.90:0/4100857336 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1da4004d50 0x7f1da40051d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009f90 tx=0x7f1db801c920 comp rx=0 tx=0).stop 2024-03-31T15:54:42.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dbf400640 1 -- 172.21.15.90:0/4100857336 reap_dead start 2024-03-31T15:54:42.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 -- 172.21.15.90:0/4100857336 shutdown_connections 2024-03-31T15:54:42.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 -- 172.21.15.90:0/4100857336 >> 172.21.15.90:0/4100857336 conn(0x7f1dc00d1f20 msgr2=0x7f1dc00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 -- 172.21.15.90:0/4100857336 shutdown_connections 2024-03-31T15:54:42.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.025+0000 7f1dc5800640 1 -- 172.21.15.90:0/4100857336 wait complete. 2024-03-31T15:54:42.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 Processor -- start 2024-03-31T15:54:42.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- start start 2024-03-31T15:54:42.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1da4004d50 0x7f1dc00c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 0x7f1dc00c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00afd90 con 0x7f1dc015e980 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc00afc10 con 0x7f1dc01591b0 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc00aff10 con 0x7f1da4004d50 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43374/0 (socket says 172.21.15.90:43374) 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1da4004d50 0x7f1dc00c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 -- 172.21.15.90:0/3338178471 learned_addr learned my addr 172.21.15.90:0/3338178471 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1db7e00640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 0x7f1dc00c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 -- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1da4004d50 msgr2=0x7f1dc00c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1da4004d50 0x7f1dc00c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 -- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 msgr2=0x7f1dc00c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 0x7f1dc00c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 -- 172.21.15.90:0/3338178471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc00b6f20 con 0x7f1dc015e980 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1db7e00640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 0x7f1dc00c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dbf400640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1db8002240 tx=0x7f1db8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b0040 con 0x7f1dc015e980 2024-03-31T15:54:42.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db80ac910 con 0x7f1dc015e980 2024-03-31T15:54:42.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8009c10 con 0x7f1dc015e980 2024-03-31T15:54:42.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc00c60e0 con 0x7f1dc015e980 2024-03-31T15:54:42.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.029+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc00c66a0 con 0x7f1dc015e980 2024-03-31T15:54:42.025 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.033+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc015f340 con 0x7f1dc015e980 2024-03-31T15:54:42.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.033+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1db80bf020 con 0x7f1dc015e980 2024-03-31T15:54:42.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.033+0000 7f1db6a00640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 0x7f1d9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.033+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1db8120eb0 con 0x7f1dc015e980 2024-03-31T15:54:42.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.033+0000 7f1dbea00640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 0x7f1d9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.037+0000 7f1dbea00640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 0x7f1d9c06a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1db0000c10 tx=0x7f1db0007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.041+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1db8016950 con 0x7f1dc015e980 2024-03-31T15:54:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.197+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1dc00b5930 con 0x7f1dc015e980 2024-03-31T15:54:42.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.197+0000 7f1db6a00640 1 -- 172.21.15.90:0/3338178471 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1db80f0fe0 con 0x7f1dc015e980 2024-03-31T15:54:42.191 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 msgr2=0x7f1d9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 0x7f1d9c06a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1db0000c10 tx=0x7f1db0007900 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 msgr2=0x7f1dc015ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1db8002240 tx=0x7f1db8009750 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 shutdown_connections 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d9c068380 0x7f1d9c06a840 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc015e980 0x7f1dc015ee00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc01591b0 0x7f1dc00c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 --2- 172.21.15.90:0/3338178471 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1da4004d50 0x7f1dc00c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 >> 172.21.15.90:0/3338178471 conn(0x7f1dc00d1f20 msgr2=0x7f1dc00b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 shutdown_connections 2024-03-31T15:54:42.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.201+0000 7f1dc5800640 1 -- 172.21.15.90:0/3338178471 wait complete. 2024-03-31T15:54:42.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:42.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 Processor -- start 2024-03-31T15:54:42.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 -- start start 2024-03-31T15:54:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffbd015e4e0 0x7ffbd006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd006ce50 0x7ffbd006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 0x7ffbd00cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd015e8c0 con 0x7ffbd015e4e0 2024-03-31T15:54:42.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd00d0ff0 con 0x7ffbd00ce7a0 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.357+0000 7ffbd6800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd00d12d0 con 0x7ffbd006ce50 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffbd015e4e0 0x7ffbd006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffbd015e4e0 0x7ffbd006c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43386/0 (socket says 172.21.15.90:43386) 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 -- 172.21.15.90:0/2830445649 learned_addr learned my addr 172.21.15.90:0/2830445649 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcea00640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd006ce50 0x7ffbd006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcfe00640 1 --2- 172.21.15.90:0/2830445649 >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 0x7ffbd00cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 -- 172.21.15.90:0/2830445649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd006ce50 msgr2=0x7ffbd006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd006ce50 0x7ffbd006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 -- 172.21.15.90:0/2830445649 >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 msgr2=0x7ffbd00cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2830445649 >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 0x7ffbd00cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 -- 172.21.15.90:0/2830445649 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd00cc710 con 0x7ffbd015e4e0 2024-03-31T15:54:42.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcfe00640 1 --2- 172.21.15.90:0/2830445649 >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 0x7ffbd00cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd006c910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0075360 tx=0x7ffbc001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305627460f00e11d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbce000640 1 -- 172.21.15.90:0/2830445649 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00b0070 con 0x7ffbd015e4e0 2024-03-31T15:54:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbce000640 1 -- 172.21.15.90:0/2830445649 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc00ab600 con 0x7ffbd015e4e0 2024-03-31T15:54:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.361+0000 7ffbce000640 1 -- 172.21.15.90:0/2830445649 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00ab9d0 con 0x7ffbd015e4e0 2024-03-31T15:54:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- 172.21.15.90:0/2830445649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 msgr2=0x7ffbd006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd006c910 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0075360 tx=0x7ffbc001c490 comp rx=0 tx=0).stop 2024-03-31T15:54:42.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- 172.21.15.90:0/2830445649 shutdown_connections 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2830445649 >> v2:172.21.15.102:3300/0 conn(0x7ffbd00ce7a0 0x7ffbd00cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd006ce50 0x7ffbd006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2830445649 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd006c910 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- 172.21.15.90:0/2830445649 >> 172.21.15.90:0/2830445649 conn(0x7ffbd00d1f20 msgr2=0x7ffbd00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- 172.21.15.90:0/2830445649 shutdown_connections 2024-03-31T15:54:42.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- 172.21.15.90:0/2830445649 wait complete. 2024-03-31T15:54:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 Processor -- start 2024-03-31T15:54:42.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- start start 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 0x7ffbd0153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 0x7ffbd0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd00cd090 con 0x7ffbd015e4e0 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd00ccf10 con 0x7ffbd006ce50 2024-03-31T15:54:42.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbd6800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd00cd210 con 0x7ffbd00ce7a0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43394/0 (socket says 172.21.15.90:43394) 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 -- 172.21.15.90:0/2698348399 learned_addr learned my addr 172.21.15.90:0/2698348399 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 0x7ffbd0153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcea00640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 0x7ffbd0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 -- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 msgr2=0x7ffbd0155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 0x7ffbd0155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 -- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 msgr2=0x7ffbd0153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 0x7ffbd0153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcfe00640 1 -- 172.21.15.90:0/2698348399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd01fb770 con 0x7ffbd015e4e0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcea00640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 0x7ffbd0155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.365+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 0x7ffbd0153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbcfe00640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009fa0 tx=0x7ffbc800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc801a070 con 0x7ffbd015e4e0 2024-03-31T15:54:42.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc8003bc0 con 0x7ffbd015e4e0 2024-03-31T15:54:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8004030 con 0x7ffbd015e4e0 2024-03-31T15:54:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd01fba50 con 0x7ffbd015e4e0 2024-03-31T15:54:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd0154d50 con 0x7ffbd015e4e0 2024-03-31T15:54:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd0154f50 con 0x7ffbd015e4e0 2024-03-31T15:54:42.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffbc801d070 con 0x7ffbd015e4e0 2024-03-31T15:54:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 0x7ffbb006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffbc8087130 con 0x7ffbd015e4e0 2024-03-31T15:54:42.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.369+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 0x7ffbb006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.373+0000 7ffbcf400640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 0x7ffbb006a840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00ad000 tx=0x7ffbc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.373+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffbc8010070 con 0x7ffbd015e4e0 2024-03-31T15:54:42.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.533+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffbd00685f0 con 0x7ffbd015e4e0 2024-03-31T15:54:42.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.533+0000 7ffbc7400640 1 -- 172.21.15.90:0/2698348399 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7ffbc8057250 con 0x7ffbd015e4e0 2024-03-31T15:54:42.526 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 msgr2=0x7ffbb006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 0x7ffbb006a840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00ad000 tx=0x7ffbc0008040 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 msgr2=0x7ffbd01fad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009fa0 tx=0x7ffbc800d560 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 shutdown_connections 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffbb0068380 0x7ffbb006a840 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffbd015e4e0 0x7ffbd01fad20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffbd00ce7a0 0x7ffbd0155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 --2- 172.21.15.90:0/2698348399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbd006ce50 0x7ffbd0153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 >> 172.21.15.90:0/2698348399 conn(0x7ffbd00d1f20 msgr2=0x7ffbd00c5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 shutdown_connections 2024-03-31T15:54:42.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.537+0000 7ffbd6800640 1 -- 172.21.15.90:0/2698348399 wait complete. 2024-03-31T15:54:42.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:42.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.709+0000 7fa831000640 1 Processor -- start 2024-03-31T15:54:42.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.709+0000 7fa831000640 1 -- start start 2024-03-31T15:54:42.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa831000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa82c1616c0 0x7fa82c0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa831000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa831000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c0cb640 con 0x7fa82c0ce830 2024-03-31T15:54:42.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa831000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa82c0cc210 con 0x7fa82c1616c0 2024-03-31T15:54:42.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa831000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa82c0ccdb0 con 0x7fa82c0afae0 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa82c1616c0 0x7fa82c0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa823e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa82c0ce830 0x7fa82c0cc6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38292/0 (socket says 172.21.15.90:38292) 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa823e00640 1 -- 172.21.15.90:0/1086761645 learned_addr learned my addr 172.21.15.90:0/1086761645 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82be00640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 -- 172.21.15.90:0/1086761645 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 msgr2=0x7fa82c0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 -- 172.21.15.90:0/1086761645 >> v1:172.21.15.90:6789/0 conn(0x7fa82c0ce830 legacy=0x7fa82c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 -- 172.21.15.90:0/1086761645 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0cb8e0 con 0x7fa82c1616c0 2024-03-31T15:54:42.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82aa00640 1 -- 172.21.15.90:0/1086761645 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784304629 0 0) 0x7fa82c0cb640 con 0x7fa82c0ce830 2024-03-31T15:54:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.713+0000 7fa82b400640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c1616c0 0x7fa82c0cf240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0097c0 tx=0x7fa81c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea5c7e23137e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa82aa00640 1 -- 172.21.15.90:0/1086761645 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0b0070 con 0x7fa82c1616c0 2024-03-31T15:54:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa82aa00640 1 -- 172.21.15.90:0/1086761645 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81c0ab600 con 0x7fa82c1616c0 2024-03-31T15:54:42.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa82aa00640 1 -- 172.21.15.90:0/1086761645 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0ab9d0 con 0x7fa82c1616c0 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- 172.21.15.90:0/1086761645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c1616c0 msgr2=0x7fa82c0cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c1616c0 0x7fa82c0cf240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0097c0 tx=0x7fa81c01c410 comp rx=0 tx=0).stop 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- 172.21.15.90:0/1086761645 shutdown_connections 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- 172.21.15.90:0/1086761645 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c1616c0 0x7fa82c0cf240 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- 172.21.15.90:0/1086761645 >> 172.21.15.90:0/1086761645 conn(0x7fa82c0d0c60 msgr2=0x7fa82c0d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- 172.21.15.90:0/1086761645 shutdown_connections 2024-03-31T15:54:42.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- 172.21.15.90:0/1086761645 wait complete. 2024-03-31T15:54:42.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 Processor -- start 2024-03-31T15:54:42.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- start start 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa82c1616c0 0x7fa82c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c161c20 con 0x7fa82c0ce830 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c161aa0 con 0x7fa82c1616c0 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa831000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa82c16ae60 con 0x7fa82c0afae0 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa823e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa823e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43402/0 (socket says 172.21.15.90:43402) 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa823e00640 1 -- 172.21.15.90:0/732194888 learned_addr learned my addr 172.21.15.90:0/732194888 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.717+0000 7fa82b400640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa82be00640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa82c1616c0 0x7fa82c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 -- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 msgr2=0x7fa82c0b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 -- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa82c1616c0 msgr2=0x7fa82c157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa82c1616c0 0x7fa82c157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 -- 172.21.15.90:0/732194888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0b6760 con 0x7fa82c0ce830 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823e00640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c480 tx=0x7fa81400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814018070 con 0x7fa82c0ce830 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa814007500 con 0x7fa82c0ce830 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c06e1c0 con 0x7fa82c0ce830 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814004a30 con 0x7fa82c0ce830 2024-03-31T15:54:42.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c06e780 con 0x7fa82c0ce830 2024-03-31T15:54:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82c068060 con 0x7fa82c0ce830 2024-03-31T15:54:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa81401b070 con 0x7fa82c0ce830 2024-03-31T15:54:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:42.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.721+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa814086060 con 0x7fa82c0ce830 2024-03-31T15:54:42.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.725+0000 7fa82b400640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:42.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.725+0000 7fa82b400640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 0x7fa80806a870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0ad000 tx=0x7fa81c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:42.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.725+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa814055bc0 con 0x7fa82c0ce830 2024-03-31T15:54:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.885+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa82c160ef0 con 0x7fa82c0ce830 2024-03-31T15:54:42.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.885+0000 7fa823400640 1 -- 172.21.15.90:0/732194888 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fa814059ec0 con 0x7fa82c0ce830 2024-03-31T15:54:42.878 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 msgr2=0x7fa80806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 0x7fa80806a870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0ad000 tx=0x7fa81c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 msgr2=0x7fa82c156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c480 tx=0x7fa81400c940 comp rx=0 tx=0).stop 2024-03-31T15:54:42.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 shutdown_connections 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa82c1616c0 0x7fa82c157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa82c0ce830 0x7fa82c156fd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 --2- 172.21.15.90:0/732194888 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa82c0afae0 0x7fa82c0b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 >> 172.21.15.90:0/732194888 conn(0x7fa82c0d0c60 msgr2=0x7fa82c06be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 shutdown_connections 2024-03-31T15:54:42.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:42.889+0000 7fa831000640 1 -- 172.21.15.90:0/732194888 wait complete. 2024-03-31T15:54:42.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-31T15:54:43.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 Processor -- start 2024-03-31T15:54:43.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 -- start start 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0dfc15e1b0 0x7f0dfc162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dfc06d7c0 0x7f0dfc15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc15f780 con 0x7f0dfc06c7b0 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc1622b0 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e03400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc15f1c0 con 0x7f0dfc15e1b0 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e01400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dfc06d7c0 0x7f0dfc15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.041+0000 7f0e01400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0dfc06d7c0 0x7f0dfc15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36968/0 (socket says 172.21.15.90:36968) 2024-03-31T15:54:43.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 -- 172.21.15.90:0/1392862140 learned_addr learned my addr 172.21.15.90:0/1392862140 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e00a00640 1 --2- 172.21.15.90:0/1392862140 >> v2:172.21.15.102:3300/0 conn(0x7f0dfc15e1b0 0x7f0dfc162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 -- 172.21.15.90:0/1392862140 >> v2:172.21.15.102:3300/0 conn(0x7f0dfc15e1b0 msgr2=0x7f0dfc162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 --2- 172.21.15.90:0/1392862140 >> v2:172.21.15.102:3300/0 conn(0x7f0dfc15e1b0 0x7f0dfc162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 -- 172.21.15.90:0/1392862140 >> v1:172.21.15.102:6789/0 conn(0x7f0dfc06c7b0 legacy=0x7f0dfc06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:43.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 -- 172.21.15.90:0/1392862140 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc15f4a0 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0e01400640 1 --2- 172.21.15.90:0/1392862140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc15e590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc067e30 tx=0x7f0df401c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57256dde580bfdce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:43.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0dfb400640 1 -- 172.21.15.90:0/1392862140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df40a57b0 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0dfb400640 1 -- 172.21.15.90:0/1392862140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df40a5950 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.045+0000 7f0dfb400640 1 -- 172.21.15.90:0/1392862140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df40a5d20 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- 172.21.15.90:0/1392862140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 msgr2=0x7f0dfc15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 --2- 172.21.15.90:0/1392862140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc15e590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc067e30 tx=0x7f0df401c8f0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- 172.21.15.90:0/1392862140 shutdown_connections 2024-03-31T15:54:43.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 --2- 172.21.15.90:0/1392862140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc15e590 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 --2- 172.21.15.90:0/1392862140 >> v2:172.21.15.102:3300/0 conn(0x7f0dfc15e1b0 0x7f0dfc162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- 172.21.15.90:0/1392862140 >> 172.21.15.90:0/1392862140 conn(0x7f0dfc09aa40 msgr2=0x7f0dfc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:43.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- 172.21.15.90:0/1392862140 shutdown_connections 2024-03-31T15:54:43.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- 172.21.15.90:0/1392862140 wait complete. 2024-03-31T15:54:43.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 Processor -- start 2024-03-31T15:54:43.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.049+0000 7f0e03400640 1 -- start start 2024-03-31T15:54:43.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dfc06c7b0 0x7f0dfc0b5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 0x7f0dfc160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0d2940 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0d27c0 con 0x7f0dfc06c7b0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc0d2ac0 con 0x7f0dfc15e1b0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36970/0 (socket says 172.21.15.90:36970) 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 -- 172.21.15.90:0/3819785479 learned_addr learned my addr 172.21.15.90:0/3819785479 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e00a00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dfc06c7b0 0x7f0dfc0b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e01400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 0x7f0dfc160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 -- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 msgr2=0x7f0dfc160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 0x7f0dfc160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 -- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dfc06c7b0 msgr2=0x7f0dfc0b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dfc06c7b0 0x7f0dfc0b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 -- 172.21.15.90:0/3819785479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc0d0d40 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e01400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 0x7f0dfc160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:43.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0dfbe00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b440 tx=0x7f0de800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8015020 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de8011890 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8011c80 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc0b5a40 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc0b6490 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.053+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc0686d0 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.057+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0de8018020 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.057+0000 7f0df8c00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 0x7f0ddc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:43.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.057+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0de8086b50 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.057+0000 7f0e00a00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 0x7f0ddc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:43.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.057+0000 7f0e00a00640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 0x7f0ddc06a840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc15a300 tx=0x7f0df0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:43.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.065+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0de801f030 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.281+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f0dfc15a050 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.993+0000 7f0df8c00640 1 -- 172.21.15.90:0/3819785479 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v74) ==== 150+0+0 (secure 0 0 0) 0x7f0de800d040 con 0x7f0dfc06d7c0 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 msgr2=0x7f0ddc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 0x7f0ddc06a840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc15a300 tx=0x7f0df0008040 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 msgr2=0x7f0dfc155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0de800b440 tx=0x7f0de800b900 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 shutdown_connections 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0ddc068380 0x7f0ddc06a840 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0dfc15e1b0 0x7f0dfc160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0dfc06d7c0 0x7f0dfc155680 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 --2- 172.21.15.90:0/3819785479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0dfc06c7b0 0x7f0dfc0b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:43.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 >> 172.21.15.90:0/3819785479 conn(0x7f0dfc09aa40 msgr2=0x7f0dfc09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 shutdown_connections 2024-03-31T15:54:43.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:43.997+0000 7f0e03400640 1 -- 172.21.15.90:0/3819785479 wait complete. 2024-03-31T15:54:44.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:54:44.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 Processor -- start 2024-03-31T15:54:44.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 -- start start 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe1b415e780 0x7fe1b40cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 0x7fe1b40d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b406c350 con 0x7fe1b40d1a00 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b406dc80 con 0x7fe1b415e780 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b8c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b4073ab0 con 0x7fe1b40ccb10 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe1b415e780 0x7fe1b40cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b2800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe1b40d1a00 0x7fe1b406cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44714/0 (socket says 172.21.15.90:44714) 2024-03-31T15:54:44.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b2800640 1 -- 172.21.15.90:0/375608004 learned_addr learned my addr 172.21.15.90:0/375608004 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:44.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1abe00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 0x7fe1b40d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1400640 1 -- 172.21.15.90:0/375608004 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225090600 0 0) 0x7fe1b406c350 con 0x7fe1b40d1a00 2024-03-31T15:54:44.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1e00640 1 -- 172.21.15.90:0/375608004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 msgr2=0x7fe1b40d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:44.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1e00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 0x7fe1b40d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1e00640 1 -- 172.21.15.90:0/375608004 >> v1:172.21.15.90:6789/0 conn(0x7fe1b40d1a00 legacy=0x7fe1b406cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1b1e00640 1 -- 172.21.15.90:0/375608004 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b406cae0 con 0x7fe1b415e780 2024-03-31T15:54:44.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.165+0000 7fe1abe00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 0x7fe1b40d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:44.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b1e00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b415e780 0x7fe1b40cc5d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40097a0 tx=0x7fe1a401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63e20b678b5a061 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b1400640 1 -- 172.21.15.90:0/375608004 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40b0070 con 0x7fe1b415e780 2024-03-31T15:54:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b1400640 1 -- 172.21.15.90:0/375608004 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a40ab600 con 0x7fe1b415e780 2024-03-31T15:54:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b1400640 1 -- 172.21.15.90:0/375608004 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40ab9d0 con 0x7fe1b415e780 2024-03-31T15:54:44.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 -- 172.21.15.90:0/375608004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b415e780 msgr2=0x7fe1b40cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b415e780 0x7fe1b40cc5d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40097a0 tx=0x7fe1a401c3f0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 -- 172.21.15.90:0/375608004 shutdown_connections 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b40ccb10 0x7fe1b40d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/375608004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b415e780 0x7fe1b40cc5d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 -- 172.21.15.90:0/375608004 >> 172.21.15.90:0/375608004 conn(0x7fe1b40cea00 msgr2=0x7fe1b40cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 -- 172.21.15.90:0/375608004 shutdown_connections 2024-03-31T15:54:44.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.169+0000 7fe1b8c00640 1 -- 172.21.15.90:0/375608004 wait complete. 2024-03-31T15:54:44.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.173+0000 7fe1b8c00640 1 Processor -- start 2024-03-31T15:54:44.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.173+0000 7fe1b8c00640 1 -- start start 2024-03-31T15:54:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1b40d1a00 0x7fe1b40b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b415e780 0x7fe1b40b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b40cc290 con 0x7fe1b40ccb10 2024-03-31T15:54:44.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b406da20 con 0x7fe1b40d1a00 2024-03-31T15:54:44.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b40cc410 con 0x7fe1b415e780 2024-03-31T15:54:44.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36984/0 (socket says 172.21.15.90:36984) 2024-03-31T15:54:44.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 -- 172.21.15.90:0/222107195 learned_addr learned my addr 172.21.15.90:0/222107195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1abe00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1b40d1a00 0x7fe1b40b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b2800640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b415e780 0x7fe1b40b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 -- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b415e780 msgr2=0x7fe1b40b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b415e780 0x7fe1b40b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 -- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1b40d1a00 msgr2=0x7fe1b40b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1b40d1a00 0x7fe1b40b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 -- 172.21.15.90:0/222107195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b40b1a10 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b1e00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe1a401aaa0 tx=0x7fe1a401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40b0050 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a40b1040 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40059a0 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b40cbb10 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b40b1cf0 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b40685f0 con 0x7fe1b40ccb10 2024-03-31T15:54:44.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.177+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe1a40be020 con 0x7fe1b40ccb10 2024-03-31T15:54:44.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.181+0000 7fe1aa000640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 0x7fe19006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.181+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe1a4120e10 con 0x7fe1b40ccb10 2024-03-31T15:54:44.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.181+0000 7fe1abe00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 0x7fe19006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.181+0000 7fe1abe00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 0x7fe19006a870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4157a70 tx=0x7fe19c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.185+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe1a4124030 con 0x7fe1b40ccb10 2024-03-31T15:54:44.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.345+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe1b406e7a0 con 0x7fe1b40ccb10 2024-03-31T15:54:44.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1aa000640 1 -- 172.21.15.90:0/222107195 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe1a4016070 con 0x7fe1b40ccb10 2024-03-31T15:54:44.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 msgr2=0x7fe19006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 0x7fe19006a870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4157a70 tx=0x7fe19c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 msgr2=0x7fe1b40c9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe1a401aaa0 tx=0x7fe1a401ac30 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 shutdown_connections 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1900683b0 0x7fe19006a870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1b415e780 0x7fe1b40b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1b40d1a00 0x7fe1b40b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 --2- 172.21.15.90:0/222107195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1b40ccb10 0x7fe1b40c9aa0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 >> 172.21.15.90:0/222107195 conn(0x7fe1b40cea00 msgr2=0x7fe1b40ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 shutdown_connections 2024-03-31T15:54:44.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.349+0000 7fe1b8c00640 1 -- 172.21.15.90:0/222107195 wait complete. 2024-03-31T15:54:44.370 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:54:44.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:54:44.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:54:44.544 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:54:44.544 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:44.544 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:44.552 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:44.553 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:44.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:44.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.789+0000 7f8f6b200640 1 Processor -- start 2024-03-31T15:54:44.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.789+0000 7f8f6b200640 1 -- start start 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f6c067330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6c067870 con 0x7f8f6c073440 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6c15c860 con 0x7f8f6c0aea30 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c15cb40 con 0x7f8f6c0af7f0 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8f6c073440 0x7f8f6c15fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44722/0 (socket says 172.21.15.90:44722) 2024-03-31T15:54:44.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6a800640 1 -- 172.21.15.90:0/1344709574 learned_addr learned my addr 172.21.15.90:0/1344709574 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:44.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f6c067330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2475203400 0 0) 0x7f8f6c067870 con 0x7f8f6c073440 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f50003530 con 0x7f8f6c073440 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3095949165 0 0) 0x7f8f50003530 con 0x7f8f6c073440 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 msgr2=0x7f8f6c067330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f6c067330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 >> v1:172.21.15.102:6789/0 conn(0x7f8f6c0aea30 legacy=0x7f8f6c072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c15ce20 con 0x7f8f6c073440 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f8f64001a00 con 0x7f8f6c073440 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 >> v1:172.21.15.90:6789/0 conn(0x7f8f6c073440 legacy=0x7f8f6c15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f500039f0 0x7f8f50003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c15cb40 con 0x7f8f500039f0 2024-03-31T15:54:44.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5000ba80 con 0x7f8f50004810 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5000bc80 con 0x7f8f6c0af7f0 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f500039f0 0x7f8f50003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f61400640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f6a800640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 -- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 msgr2=0x7f8f500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c0af7f0 0x7f8f500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 -- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f50004810 msgr2=0x7f8f50004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.793+0000 7f8f69e00640 1 -- 172.21.15.90:0/1344709574 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f50003530 con 0x7f8f500039f0 2024-03-31T15:54:44.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f69e00640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f500039f0 0x7f8f50003e10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009ab0 tx=0x7f8f5c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4aa3dffd6fc2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0096b0 con 0x7f8f500039f0 2024-03-31T15:54:44.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0a6410 con 0x7f8f500039f0 2024-03-31T15:54:44.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f69400640 1 -- 172.21.15.90:0/1344709574 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a6800 con 0x7f8f500039f0 2024-03-31T15:54:44.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f6b200640 1 -- 172.21.15.90:0/1344709574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f500039f0 msgr2=0x7f8f50003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f6b200640 1 --2- 172.21.15.90:0/1344709574 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f500039f0 0x7f8f50003e10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009ab0 tx=0x7f8f5c01ca30 comp rx=0 tx=0).stop 2024-03-31T15:54:44.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.797+0000 7f8f6a800640 1 -- 172.21.15.90:0/1344709574 reap_dead start 2024-03-31T15:54:44.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- 172.21.15.90:0/1344709574 shutdown_connections 2024-03-31T15:54:44.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- 172.21.15.90:0/1344709574 >> 172.21.15.90:0/1344709574 conn(0x7f8f6c0b5130 msgr2=0x7f8f6c0b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:44.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- 172.21.15.90:0/1344709574 shutdown_connections 2024-03-31T15:54:44.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- 172.21.15.90:0/1344709574 wait complete. 2024-03-31T15:54:44.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 Processor -- start 2024-03-31T15:54:44.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- start start 2024-03-31T15:54:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 0x7f8f6c14a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c15d920 0x7f8f6c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c0afd50 con 0x7f8f6c1582f0 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6c0afbd0 con 0x7f8f500039f0 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6c15c430 con 0x7f8f6c15d920 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37002/0 (socket says 172.21.15.90:37002) 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 -- 172.21.15.90:0/3504217910 learned_addr learned my addr 172.21.15.90:0/3504217910 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f69e00640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 0x7f8f6c14a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f61400640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c15d920 0x7f8f6c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 -- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c15d920 msgr2=0x7f8f6c15dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c15d920 0x7f8f6c15dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 -- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 msgr2=0x7f8f6c14a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 0x7f8f6c14a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 -- 172.21.15.90:0/3504217910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c14ad80 con 0x7f8f6c1582f0 2024-03-31T15:54:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f69e00640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 0x7f8f6c14a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.801+0000 7f8f6a800640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0d13b0 tx=0x7f8f64007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6400ab20 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f64019070 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f640133f0 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6c1f6a70 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6c1f7030 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f6c15e2e0 con 0x7f8f6c1582f0 2024-03-31T15:54:44.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8f64013590 con 0x7f8f6c1582f0 2024-03-31T15:54:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 0x7f8f4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8f640871d0 con 0x7f8f6c1582f0 2024-03-31T15:54:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.805+0000 7f8f69e00640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 0x7f8f4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:44.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.809+0000 7f8f69e00640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 0x7f8f4806a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0ad000 tx=0x7f8f5c0b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:44.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.809+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8f64057390 con 0x7f8f6c1582f0 2024-03-31T15:54:44.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.969+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f6c068d00 con 0x7f8f6c1582f0 2024-03-31T15:54:44.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.969+0000 7f8f62a00640 1 -- 172.21.15.90:0/3504217910 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f8f64056c90 con 0x7f8f6c1582f0 2024-03-31T15:54:44.964 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:44.964 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 msgr2=0x7f8f4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 0x7f8f4806a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0ad000 tx=0x7f8f5c0b2040 comp rx=0 tx=0).stop 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 msgr2=0x7f8f6c158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0d13b0 tx=0x7f8f64007a70 comp rx=0 tx=0).stop 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 shutdown_connections 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f48068380 0x7f8f4806a840 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f6c1582f0 0x7f8f6c158770 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f6c15d920 0x7f8f6c15dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 --2- 172.21.15.90:0/3504217910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f500039f0 0x7f8f6c14a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 >> 172.21.15.90:0/3504217910 conn(0x7f8f6c0b5130 msgr2=0x7f8f6c15f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 shutdown_connections 2024-03-31T15:54:44.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:44.973+0000 7f8f6b200640 1 -- 172.21.15.90:0/3504217910 wait complete. 2024-03-31T15:54:44.985 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:44.986 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 Processor -- start 2024-03-31T15:54:45.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 -- start start 2024-03-31T15:54:45.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f542c15e4e0 0x7f542c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c15e8c0 con 0x7f542c0ce7a0 2024-03-31T15:54:45.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f542c0d0ff0 con 0x7f542c15e4e0 2024-03-31T15:54:45.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5433c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f542c0d12d0 con 0x7f542c06ce50 2024-03-31T15:54:45.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5431c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f542c0ce7a0 0x7f542c0cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44730/0 (socket says 172.21.15.90:44730) 2024-03-31T15:54:45.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5431c00640 1 -- 172.21.15.90:0/1830980736 learned_addr learned my addr 172.21.15.90:0/1830980736 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f5431200640 1 --2- 172.21.15.90:0/1830980736 >> v2:172.21.15.102:3300/0 conn(0x7f542c15e4e0 0x7f542c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.133+0000 7f542be00640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542b400640 1 -- 172.21.15.90:0/1830980736 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 517357081 0 0) 0x7f542c15e8c0 con 0x7f542c0ce7a0 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542b400640 1 -- 172.21.15.90:0/1830980736 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5414003530 con 0x7f542c0ce7a0 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f5431200640 1 -- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 msgr2=0x7f542c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f5431200640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f5431200640 1 -- 172.21.15.90:0/1830980736 >> v1:172.21.15.90:6789/0 conn(0x7f542c0ce7a0 legacy=0x7f542c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f5431200640 1 -- 172.21.15.90:0/1830980736 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c0cc710 con 0x7f542c15e4e0 2024-03-31T15:54:45.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542be00640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:45.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f5431200640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c06c910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f54200097c0 tx=0x7f542001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7467faa0dbedc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542b400640 1 -- 172.21.15.90:0/1830980736 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200b0070 con 0x7f542c15e4e0 2024-03-31T15:54:45.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542b400640 1 -- 172.21.15.90:0/1830980736 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54200ab600 con 0x7f542c15e4e0 2024-03-31T15:54:45.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.137+0000 7f542b400640 1 -- 172.21.15.90:0/1830980736 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200ab9b0 con 0x7f542c15e4e0 2024-03-31T15:54:45.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 msgr2=0x7f542c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c06c910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f54200097c0 tx=0x7f542001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- 172.21.15.90:0/1830980736 shutdown_connections 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- 172.21.15.90:0/1830980736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c06c910 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- 172.21.15.90:0/1830980736 >> 172.21.15.90:0/1830980736 conn(0x7f542c0d1f20 msgr2=0x7f542c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- 172.21.15.90:0/1830980736 shutdown_connections 2024-03-31T15:54:45.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- 172.21.15.90:0/1830980736 wait complete. 2024-03-31T15:54:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 Processor -- start 2024-03-31T15:54:45.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- start start 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c0cb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f542c0d0d90 con 0x7f542c0ce7a0 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f542c0d0c10 con 0x7f542c15e4e0 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5433c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f542c16b840 con 0x7f542c06ce50 2024-03-31T15:54:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37006/0 (socket says 172.21.15.90:37006) 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 -- 172.21.15.90:0/3107341055 learned_addr learned my addr 172.21.15.90:0/3107341055 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5431c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c0cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5431200640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 -- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 msgr2=0x7f542c1565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c1565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 -- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 msgr2=0x7f542c0cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c0cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 -- 172.21.15.90:0/3107341055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c154e60 con 0x7f542c0ce7a0 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5431200640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f5431c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.141+0000 7f542be00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f541800e980 tx=0x7f541800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418017070 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f541800ca00 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c0c9f70 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541800cdd0 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c0ca530 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f542c0685f0 con 0x7f542c0ce7a0 2024-03-31T15:54:45.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f541801a070 con 0x7f542c0ce7a0 2024-03-31T15:54:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 0x7f540c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5418086c90 con 0x7f542c0ce7a0 2024-03-31T15:54:45.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.145+0000 7f5431200640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 0x7f540c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.149+0000 7f5431200640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 0x7f540c06a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f54200ad000 tx=0x7f54200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.149+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5418056e40 con 0x7f542c0ce7a0 2024-03-31T15:54:45.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.309+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f542c0c66c0 con 0x7f542c0ce7a0 2024-03-31T15:54:45.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.309+0000 7f5428c00640 1 -- 172.21.15.90:0/3107341055 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f5418056740 con 0x7f542c0ce7a0 2024-03-31T15:54:45.302 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:45.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 msgr2=0x7f540c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 0x7f540c06a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f54200ad000 tx=0x7f54200a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:45.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 msgr2=0x7f542c154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f541800e980 tx=0x7f541800ee40 comp rx=0 tx=0).stop 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 shutdown_connections 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f540c0683b0 0x7f540c06a870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f542c15e4e0 0x7f542c0cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f542c0ce7a0 0x7f542c154920 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 --2- 172.21.15.90:0/3107341055 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f542c06ce50 0x7f542c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 >> 172.21.15.90:0/3107341055 conn(0x7f542c0d1f20 msgr2=0x7f542c070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 shutdown_connections 2024-03-31T15:54:45.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.313+0000 7f5433c00640 1 -- 172.21.15.90:0/3107341055 wait complete. 2024-03-31T15:54:45.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:45.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 Processor -- start 2024-03-31T15:54:45.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 -- start start 2024-03-31T15:54:45.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec4c15c320 0x7fec4c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec4c0cd470 0x7fec4c0cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fec4c0aed70 con 0x7fec4c0cd470 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fec4c0cd900 con 0x7fec4c15c320 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec50e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fec4c0cefc0 con 0x7fec4c0d2b20 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec4c0cd470 0x7fec4c0cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec4c0cd470 0x7fec4c0cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37022/0 (socket says 172.21.15.90:37022) 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 -- 172.21.15.90:0/414387025 learned_addr learned my addr 172.21.15.90:0/414387025 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec49e00640 1 --2- 172.21.15.90:0/414387025 >> v2:172.21.15.102:3300/0 conn(0x7fec4c15c320 0x7fec4c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec49400640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 -- 172.21.15.90:0/414387025 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 msgr2=0x7fec4c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 -- 172.21.15.90:0/414387025 >> v2:172.21.15.102:3300/0 conn(0x7fec4c15c320 msgr2=0x7fec4c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 --2- 172.21.15.90:0/414387025 >> v2:172.21.15.102:3300/0 conn(0x7fec4c15c320 0x7fec4c0cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.477+0000 7fec4a800640 1 -- 172.21.15.90:0/414387025 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c0cf2a0 con 0x7fec4c0cd470 2024-03-31T15:54:45.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec4a800640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c0cd470 0x7fec4c0cfd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0739b0 tx=0x7fec4401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496c048eff8fd03f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec48a00640 1 -- 172.21.15.90:0/414387025 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440a5690 con 0x7fec4c0cd470 2024-03-31T15:54:45.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec48a00640 1 -- 172.21.15.90:0/414387025 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec440a5830 con 0x7fec4c0cd470 2024-03-31T15:54:45.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec48a00640 1 -- 172.21.15.90:0/414387025 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440a5c00 con 0x7fec4c0cd470 2024-03-31T15:54:45.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 -- 172.21.15.90:0/414387025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c0cd470 msgr2=0x7fec4c0cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c0cd470 0x7fec4c0cfd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0739b0 tx=0x7fec4401c960 comp rx=0 tx=0).stop 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 -- 172.21.15.90:0/414387025 shutdown_connections 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c0cd470 0x7fec4c0cfd00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 --2- 172.21.15.90:0/414387025 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 --2- 172.21.15.90:0/414387025 >> v2:172.21.15.102:3300/0 conn(0x7fec4c15c320 0x7fec4c0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 -- 172.21.15.90:0/414387025 >> 172.21.15.90:0/414387025 conn(0x7fec4c0d1f20 msgr2=0x7fec4c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 -- 172.21.15.90:0/414387025 shutdown_connections 2024-03-31T15:54:45.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.481+0000 7fec50e00640 1 -- 172.21.15.90:0/414387025 wait complete. 2024-03-31T15:54:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 Processor -- start 2024-03-31T15:54:45.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- start start 2024-03-31T15:54:45.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec4c0cd470 0x7fec4c0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c0d0e20 con 0x7fec4c15c320 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c0d0ca0 con 0x7fec4c0cd470 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fec4c0af1e0 con 0x7fec4c0d2b20 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37036/0 (socket says 172.21.15.90:37036) 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 -- 172.21.15.90:0/1536947833 learned_addr learned my addr 172.21.15.90:0/1536947833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec49400640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec49e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec4c0cd470 0x7fec4c0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 -- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 msgr2=0x7fec4c0b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 -- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec4c0cd470 msgr2=0x7fec4c0c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec4c0cd470 0x7fec4c0c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 -- 172.21.15.90:0/1536947833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c154430 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec4a800640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fec4401b010 tx=0x7fec44019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440bc020 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec440b8670 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44009510 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c0c8890 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.485+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c1f6650 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.489+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c0685f0 con 0x7fec4c15c320 2024-03-31T15:54:45.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.489+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fec440bf020 con 0x7fec4c15c320 2024-03-31T15:54:45.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.489+0000 7fec42000640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 0x7fec2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.489+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fec44121c90 con 0x7fec4c15c320 2024-03-31T15:54:45.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.489+0000 7fec49e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 0x7fec2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.493+0000 7fec49e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 0x7fec2c06a840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fec4c15b280 tx=0x7fec3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.497+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fec440c6030 con 0x7fec4c15c320 2024-03-31T15:54:45.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.649+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fec4c15f1c0 con 0x7fec4c15c320 2024-03-31T15:54:45.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec42000640 1 -- 172.21.15.90:0/1536947833 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fec440165e0 con 0x7fec4c15c320 2024-03-31T15:54:45.644 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 msgr2=0x7fec2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 0x7fec2c06a840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fec4c15b280 tx=0x7fec3c008040 comp rx=0 tx=0).stop 2024-03-31T15:54:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 msgr2=0x7fec4c0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fec4401b010 tx=0x7fec44019700 comp rx=0 tx=0).stop 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 shutdown_connections 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec2c068380 0x7fec2c06a840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec4c15c320 0x7fec4c0b3470 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec4c0d2b20 0x7fec4c0b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 --2- 172.21.15.90:0/1536947833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec4c0cd470 0x7fec4c0c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 >> 172.21.15.90:0/1536947833 conn(0x7fec4c0d1f20 msgr2=0x7fec4c15e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 shutdown_connections 2024-03-31T15:54:45.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.653+0000 7fec50e00640 1 -- 172.21.15.90:0/1536947833 wait complete. 2024-03-31T15:54:45.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 Processor -- start 2024-03-31T15:54:45.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 -- start start 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f741c06d020 0x7f741c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 0x7f741c0cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741c15e8c0 con 0x7f741c15e4e0 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f741c0d0ff0 con 0x7f741c06d020 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.809+0000 7f7423200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f741c0d12d0 con 0x7f741c06d490 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f741be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f741c15e4e0 0x7f741c06c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44740/0 (socket says 172.21.15.90:44740) 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f741be00640 1 -- 172.21.15.90:0/1824487871 learned_addr learned my addr 172.21.15.90:0/1824487871 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:45.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 --2- 172.21.15.90:0/1824487871 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 0x7f741c0cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7413e00640 1 --2- 172.21.15.90:0/1824487871 >> v2:172.21.15.102:3300/0 conn(0x7f741c06d020 0x7f741c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f741b400640 1 -- 172.21.15.90:0/1824487871 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573888574 0 0) 0x7f741c15e8c0 con 0x7f741c15e4e0 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f741b400640 1 -- 172.21.15.90:0/1824487871 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7400003530 con 0x7f741c15e4e0 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 -- 172.21.15.90:0/1824487871 >> v2:172.21.15.102:3300/0 conn(0x7f741c06d020 msgr2=0x7f741c0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 --2- 172.21.15.90:0/1824487871 >> v2:172.21.15.102:3300/0 conn(0x7f741c06d020 0x7f741c0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 -- 172.21.15.90:0/1824487871 >> v1:172.21.15.90:6789/0 conn(0x7f741c15e4e0 legacy=0x7f741c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 -- 172.21.15.90:0/1824487871 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c0cc710 con 0x7f741c06d490 2024-03-31T15:54:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.813+0000 7f7421200640 1 --2- 172.21.15.90:0/1824487871 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 0x7f741c0cd450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7414009dc0 tx=0x7f741401c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b8fd4b0f185cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f741b400640 1 -- 172.21.15.90:0/1824487871 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140b2070 con 0x7f741c06d490 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f741b400640 1 -- 172.21.15.90:0/1824487871 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74140a5780 con 0x7f741c06d490 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f741b400640 1 -- 172.21.15.90:0/1824487871 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140a5b50 con 0x7f741c06d490 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- 172.21.15.90:0/1824487871 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 msgr2=0x7f741c0cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- 172.21.15.90:0/1824487871 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 0x7f741c0cd450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7414009dc0 tx=0x7f741401c8c0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- 172.21.15.90:0/1824487871 shutdown_connections 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- 172.21.15.90:0/1824487871 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d490 0x7f741c0cd450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- 172.21.15.90:0/1824487871 >> v2:172.21.15.102:3300/0 conn(0x7f741c06d020 0x7f741c0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- 172.21.15.90:0/1824487871 >> 172.21.15.90:0/1824487871 conn(0x7f741c0d1f20 msgr2=0x7f741c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- 172.21.15.90:0/1824487871 shutdown_connections 2024-03-31T15:54:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- 172.21.15.90:0/1824487871 wait complete. 2024-03-31T15:54:45.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 Processor -- start 2024-03-31T15:54:45.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- start start 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d020 0x7f741c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f741c15e4e0 0x7f741c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f741c0d0d90 con 0x7f741c06d490 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f741c0d0c10 con 0x7f741c15e4e0 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7423200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f741c16b030 con 0x7f741c06d020 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7413e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7413e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37042/0 (socket says 172.21.15.90:37042) 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7413e00640 1 -- 172.21.15.90:0/1134036183 learned_addr learned my addr 172.21.15.90:0/1134036183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f741be00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d020 0x7f741c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.817+0000 7f7421200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f741c15e4e0 0x7f741c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 -- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d020 msgr2=0x7f741c154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d020 0x7f741c154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 -- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f741c15e4e0 msgr2=0x7f741c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f741c15e4e0 0x7f741c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 -- 172.21.15.90:0/1134036183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c155d40 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7413e00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f740400e930 tx=0x7f740400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404017070 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f740400ca00 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741c156020 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740400cdb0 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741c1565e0 con 0x7f741c06d490 2024-03-31T15:54:45.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741c0b7d20 con 0x7f741c06d490 2024-03-31T15:54:45.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f740401a070 con 0x7f741c06d490 2024-03-31T15:54:45.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 0x7f73f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:45.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.821+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f74040870d0 con 0x7f741c06d490 2024-03-31T15:54:45.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.825+0000 7f741be00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 0x7f73f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:45.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.825+0000 7f741be00640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 0x7f73f806a840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f740c007fd0 tx=0x7f740c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:45.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.825+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7404056cf0 con 0x7f741c06d490 2024-03-31T15:54:45.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.981+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f741c0685f0 con 0x7f741c06d490 2024-03-31T15:54:45.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7418c00640 1 -- 172.21.15.90:0/1134036183 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f740405aff0 con 0x7f741c06d490 2024-03-31T15:54:45.977 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 msgr2=0x7f73f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 0x7f73f806a840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f740c007fd0 tx=0x7f740c009040 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 msgr2=0x7f741c0c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f740400e930 tx=0x7f740400edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 shutdown_connections 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f73f8068380 0x7f73f806a840 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f741c15e4e0 0x7f741c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f741c06d490 0x7f741c0c9e70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 --2- 172.21.15.90:0/1134036183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f741c06d020 0x7f741c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 >> 172.21.15.90:0/1134036183 conn(0x7f741c0d1f20 msgr2=0x7f741c0c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 shutdown_connections 2024-03-31T15:54:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:45.985+0000 7f7423200640 1 -- 172.21.15.90:0/1134036183 wait complete. 2024-03-31T15:54:45.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:46.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 Processor -- start 2024-03-31T15:54:46.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 -- start start 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd54015c320 0x7fd5400cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd5400ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5400cd470 0x7fd5400cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd5400aed70 con 0x7fd5400cd470 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd5400cd900 con 0x7fd54015c320 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd546200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd5400cefc0 con 0x7fd5400d2b20 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5400cd470 0x7fd5400cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5400cd470 0x7fd5400cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37048/0 (socket says 172.21.15.90:37048) 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 -- 172.21.15.90:0/2913266240 learned_addr learned my addr 172.21.15.90:0/2913266240 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53f400640 1 --2- 172.21.15.90:0/2913266240 >> v2:172.21.15.102:3300/0 conn(0x7fd54015c320 0x7fd5400cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd537e00640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd5400ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 -- 172.21.15.90:0/2913266240 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 msgr2=0x7fd5400ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd5400ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 -- 172.21.15.90:0/2913266240 >> v2:172.21.15.102:3300/0 conn(0x7fd54015c320 msgr2=0x7fd5400cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:46.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 --2- 172.21.15.90:0/2913266240 >> v2:172.21.15.102:3300/0 conn(0x7fd54015c320 0x7fd5400cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.145+0000 7fd53fe00640 1 -- 172.21.15.90:0/2913266240 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5400cf2a0 con 0x7fd5400cd470 2024-03-31T15:54:46.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd53fe00640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd5400cfd00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd5400739b0 tx=0x7fd53801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614bb931dfa988da server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd53ea00640 1 -- 172.21.15.90:0/2913266240 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5380a5690 con 0x7fd5400cd470 2024-03-31T15:54:46.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd53ea00640 1 -- 172.21.15.90:0/2913266240 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5380a5830 con 0x7fd5400cd470 2024-03-31T15:54:46.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd53ea00640 1 -- 172.21.15.90:0/2913266240 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5380a5c00 con 0x7fd5400cd470 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 -- 172.21.15.90:0/2913266240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 msgr2=0x7fd5400cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd5400cfd00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd5400739b0 tx=0x7fd53801c960 comp rx=0 tx=0).stop 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 -- 172.21.15.90:0/2913266240 shutdown_connections 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd5400cfd00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 --2- 172.21.15.90:0/2913266240 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd5400ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 --2- 172.21.15.90:0/2913266240 >> v2:172.21.15.102:3300/0 conn(0x7fd54015c320 0x7fd5400cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 -- 172.21.15.90:0/2913266240 >> 172.21.15.90:0/2913266240 conn(0x7fd5400d1f20 msgr2=0x7fd5400d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 -- 172.21.15.90:0/2913266240 shutdown_connections 2024-03-31T15:54:46.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.149+0000 7fd546200640 1 -- 172.21.15.90:0/2913266240 wait complete. 2024-03-31T15:54:46.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 Processor -- start 2024-03-31T15:54:46.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- start start 2024-03-31T15:54:46.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd540158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd54015c320 0x7fd540156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd5400d0e20 con 0x7fd5400cd470 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5400d0ca0 con 0x7fd54015c320 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd5400af1e0 con 0x7fd5400d2b20 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37064/0 (socket says 172.21.15.90:37064) 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 -- 172.21.15.90:0/717977084 learned_addr learned my addr 172.21.15.90:0/717977084 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53fe00640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd54015c320 0x7fd540156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd537e00640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd540158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 -- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 msgr2=0x7fd540158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd540158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 -- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd54015c320 msgr2=0x7fd540156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd54015c320 0x7fd540156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 -- 172.21.15.90:0/717977084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd540155cc0 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd53f400640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d5f0 tx=0x7fd53000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300125f0 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd530019070 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300129b0 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd540155fa0 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd540156560 con 0x7fd5400cd470 2024-03-31T15:54:46.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.153+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5400685f0 con 0x7fd5400cd470 2024-03-31T15:54:46.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.157+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd530012b50 con 0x7fd5400cd470 2024-03-31T15:54:46.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.157+0000 7fd537400640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 0x7fd51c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.157+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd5300870d0 con 0x7fd5400cd470 2024-03-31T15:54:46.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.157+0000 7fd537e00640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 0x7fd51c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.157+0000 7fd537e00640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 0x7fd51c06a840 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd5401521f0 tx=0x7fd528008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.161+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd530056ba0 con 0x7fd5400cd470 2024-03-31T15:54:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.321+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd5400cc350 con 0x7fd5400cd470 2024-03-31T15:54:46.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.321+0000 7fd537400640 1 -- 172.21.15.90:0/717977084 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fd53005aea0 con 0x7fd5400cd470 2024-03-31T15:54:46.316 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 msgr2=0x7fd51c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 0x7fd51c06a840 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd5401521f0 tx=0x7fd528008040 comp rx=0 tx=0).stop 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 msgr2=0x7fd540164fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd53000d5f0 tx=0x7fd53000dab0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 shutdown_connections 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd51c068380 0x7fd51c06a840 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd54015c320 0x7fd540156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5400d2b20 0x7fd540158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 --2- 172.21.15.90:0/717977084 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5400cd470 0x7fd540164fc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 >> 172.21.15.90:0/717977084 conn(0x7fd5400d1f20 msgr2=0x7fd540154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 shutdown_connections 2024-03-31T15:54:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.325+0000 7fd546200640 1 -- 172.21.15.90:0/717977084 wait complete. 2024-03-31T15:54:46.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:46.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 Processor -- start 2024-03-31T15:54:46.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 -- start start 2024-03-31T15:54:46.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f76c4164be0 0x7f76c4166fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c40d0ea0 con 0x7f76c40ce910 2024-03-31T15:54:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c40cd460 con 0x7f76c415d240 2024-03-31T15:54:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76ca000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f76c40cd220 con 0x7f76c4164be0 2024-03-31T15:54:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76c3400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f76c40ce910 0x7f76c40cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44744/0 (socket says 172.21.15.90:44744) 2024-03-31T15:54:46.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.481+0000 7f76c3400640 1 -- 172.21.15.90:0/3323123746 learned_addr learned my addr 172.21.15.90:0/3323123746 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c3e00640 1 --2- 172.21.15.90:0/3323123746 >> v2:172.21.15.102:3300/0 conn(0x7f76c4164be0 0x7f76c4166fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251421105 0 0) 0x7f76c40d0ea0 con 0x7f76c40ce910 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76a8003530 con 0x7f76c40ce910 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922815944 0 0) 0x7f76c40cd460 con 0x7f76c415d240 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76c40d0ea0 con 0x7f76c415d240 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1285411926 0 0) 0x7f76a8003530 con 0x7f76c40ce910 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 >> v2:172.21.15.102:3300/0 conn(0x7f76c4164be0 msgr2=0x7f76c4166fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 --2- 172.21.15.90:0/3323123746 >> v2:172.21.15.102:3300/0 conn(0x7f76c4164be0 0x7f76c4166fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 >> v1:172.21.15.102:6789/0 conn(0x7f76c415d240 legacy=0x7f76c40ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c40cd740 con 0x7f76c40ce910 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f76b4001680 con 0x7f76c40ce910 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 >> v1:172.21.15.90:6789/0 conn(0x7f76c40ce910 legacy=0x7f76c40cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76a8003ba0 0x7f76a8003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 0x7f76a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8005220 0x7f76a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f76c40cd220 con 0x7f76a8004810 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76a800c150 con 0x7f76a8003ba0 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f76a800c350 con 0x7f76a8005220 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 0x7f76a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c3400640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76a8003ba0 0x7f76a8003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c3e00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8005220 0x7f76a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 -- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8005220 msgr2=0x7f76a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8005220 0x7f76a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 -- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76a8003ba0 msgr2=0x7f76a8003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:46.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76a8003ba0 0x7f76a8003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76bbe00640 1 -- 172.21.15.90:0/3323123746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a8003530 con 0x7f76a8004810 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.485+0000 7f76c3e00640 1 -- 172.21.15.90:0/3323123746 reap_dead start 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76bbe00640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 0x7f76a8004c90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0018d0 tx=0x7f76ac01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44136af51d59963f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac0b0070 con 0x7f76a8004810 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76ac0050d0 con 0x7f76a8004810 2024-03-31T15:54:46.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76c2a00640 1 -- 172.21.15.90:0/3323123746 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ac001520 con 0x7f76a8004810 2024-03-31T15:54:46.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 msgr2=0x7f76a8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 0x7f76a8004c90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0018d0 tx=0x7f76ac01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- 172.21.15.90:0/3323123746 shutdown_connections 2024-03-31T15:54:46.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 --2- 172.21.15.90:0/3323123746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76a8004810 0x7f76a8004c90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- 172.21.15.90:0/3323123746 >> 172.21.15.90:0/3323123746 conn(0x7f76c408bfd0 msgr2=0x7f76c40d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:46.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- 172.21.15.90:0/3323123746 shutdown_connections 2024-03-31T15:54:46.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- 172.21.15.90:0/3323123746 wait complete. 2024-03-31T15:54:46.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 Processor -- start 2024-03-31T15:54:46.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 -- start start 2024-03-31T15:54:46.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.489+0000 7f76ca000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 0x7f76c40b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 0x7f76c40c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f76c40cd060 con 0x7f76c4162c10 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76c40ccee0 con 0x7f76c4150550 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f76c40cf100 con 0x7f76a8004810 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 0x7f76c40b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 0x7f76c40b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:60780/0 (socket says 172.21.15.90:60780) 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3400640 1 -- 172.21.15.90:0/1648997882 learned_addr learned my addr 172.21.15.90:0/1648997882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37086/0 (socket says 172.21.15.90:37086) 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76bbe00640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 0x7f76c40c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 -- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 msgr2=0x7f76c40b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 0x7f76c40b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 -- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 msgr2=0x7f76c40c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 0x7f76c40c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 -- 172.21.15.90:0/1648997882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c41509d0 con 0x7f76c4162c10 2024-03-31T15:54:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76bbe00640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 0x7f76c40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:46.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76c3e00640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f76c40cf820 tx=0x7f76bc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc011e90 con 0x7f76c4162c10 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc01c070 con 0x7f76c4162c10 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc003be0 con 0x7f76c4162c10 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c40b5150 con 0x7f76c4162c10 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c40b5710 con 0x7f76c4162c10 2024-03-31T15:54:46.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.493+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c4068660 con 0x7f76c4162c10 2024-03-31T15:54:46.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.497+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f76bc004030 con 0x7f76c4162c10 2024-03-31T15:54:46.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.497+0000 7f76bb400640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 0x7f76a006aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.497+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f76bc087530 con 0x7f76c4162c10 2024-03-31T15:54:46.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.497+0000 7f76c3400640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 0x7f76a006aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.497+0000 7f76c3400640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 0x7f76a006aa10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f76b4007c70 tx=0x7f76b4001050 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.501+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f76bc010040 con 0x7f76c4162c10 2024-03-31T15:54:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f76c40ce6f0 con 0x7f76c4162c10 2024-03-31T15:54:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76bb400640 1 -- 172.21.15.90:0/1648997882 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f76bc057780 con 0x7f76c4162c10 2024-03-31T15:54:46.662 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 msgr2=0x7f76a006aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 0x7f76a006aa10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f76b4007c70 tx=0x7f76b4001050 comp rx=0 tx=0).stop 2024-03-31T15:54:46.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 msgr2=0x7f76c4163090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f76c40cf820 tx=0x7f76bc00f580 comp rx=0 tx=0).stop 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 shutdown_connections 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f76a0068550 0x7f76a006aa10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f76c4162c10 0x7f76c4163090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76c4150550 0x7f76c40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 --2- 172.21.15.90:0/1648997882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f76a8004810 0x7f76c40b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 >> 172.21.15.90:0/1648997882 conn(0x7f76c408bfd0 msgr2=0x7f76c4070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 shutdown_connections 2024-03-31T15:54:46.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.669+0000 7f76ca000640 1 -- 172.21.15.90:0/1648997882 wait complete. 2024-03-31T15:54:46.683 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:46.683 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:46.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:46.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 Processor -- start 2024-03-31T15:54:46.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 -- start start 2024-03-31T15:54:46.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97c815b010 0x7f97c815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c815eb10 con 0x7f97c806d2a0 2024-03-31T15:54:46.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97c80708b0 con 0x7f97c80723c0 2024-03-31T15:54:46.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97ce200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f97c815b3f0 con 0x7f97c815b010 2024-03-31T15:54:46.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97c7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f97c806d2a0 0x7f97c80cfc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44750/0 (socket says 172.21.15.90:44750) 2024-03-31T15:54:46.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97c7e00640 1 -- 172.21.15.90:0/3591370316 learned_addr learned my addr 172.21.15.90:0/3591370316 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:46.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.877+0000 7f97c7400640 1 --2- 172.21.15.90:0/3591370316 >> v2:172.21.15.102:3300/0 conn(0x7f97c815b010 0x7f97c815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c6a00640 1 -- 172.21.15.90:0/3591370316 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762070566 0 0) 0x7f97c815eb10 con 0x7f97c806d2a0 2024-03-31T15:54:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c6a00640 1 -- 172.21.15.90:0/3591370316 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97ac005740 con 0x7f97c806d2a0 2024-03-31T15:54:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c7400640 1 -- 172.21.15.90:0/3591370316 >> v1:172.21.15.102:6789/0 conn(0x7f97c80723c0 legacy=0x7f97c806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c7400640 1 -- 172.21.15.90:0/3591370316 >> v1:172.21.15.90:6789/0 conn(0x7f97c806d2a0 legacy=0x7f97c80cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c7400640 1 -- 172.21.15.90:0/3591370316 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c8072f30 con 0x7f97c815b010 2024-03-31T15:54:46.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c7400640 1 --2- 172.21.15.90:0/3591370316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c815b010 0x7f97c815f5b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f97b80097c0 tx=0x7f97b801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e962eea7b0a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c6a00640 1 -- 172.21.15.90:0/3591370316 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b80a7070 con 0x7f97c815b010 2024-03-31T15:54:46.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c6a00640 1 -- 172.21.15.90:0/3591370316 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97b80b0070 con 0x7f97c815b010 2024-03-31T15:54:46.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.881+0000 7f97c6a00640 1 -- 172.21.15.90:0/3591370316 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b80ab890 con 0x7f97c815b010 2024-03-31T15:54:46.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- 172.21.15.90:0/3591370316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c815b010 msgr2=0x7f97c815f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 --2- 172.21.15.90:0/3591370316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c815b010 0x7f97c815f5b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f97b80097c0 tx=0x7f97b801c470 comp rx=0 tx=0).stop 2024-03-31T15:54:46.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- 172.21.15.90:0/3591370316 shutdown_connections 2024-03-31T15:54:46.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 --2- 172.21.15.90:0/3591370316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c815b010 0x7f97c815f5b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- 172.21.15.90:0/3591370316 >> 172.21.15.90:0/3591370316 conn(0x7f97c80d13a0 msgr2=0x7f97c80d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:46.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- 172.21.15.90:0/3591370316 shutdown_connections 2024-03-31T15:54:46.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- 172.21.15.90:0/3591370316 wait complete. 2024-03-31T15:54:46.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 Processor -- start 2024-03-31T15:54:46.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- start start 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c806d2a0 0x7f97c80b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97c80723c0 0x7f97c8150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97c806b5f0 con 0x7f97c815b010 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97c8072bb0 con 0x7f97c806d2a0 2024-03-31T15:54:46.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97ce200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f97c806b770 con 0x7f97c80723c0 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37098/0 (socket says 172.21.15.90:37098) 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 -- 172.21.15.90:0/970054490 learned_addr learned my addr 172.21.15.90:0/970054490 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7400640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c806d2a0 0x7f97c80b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97bfe00640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97c80723c0 0x7f97c8150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 -- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97c80723c0 msgr2=0x7f97c8150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97c80723c0 0x7f97c8150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 -- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c806d2a0 msgr2=0x7f97c80b6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c806d2a0 0x7f97c80b6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.885+0000 7f97c7e00640 1 -- 172.21.15.90:0/970054490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c8151690 con 0x7f97c815b010 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97c7e00640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f97c000cbc0 tx=0x7f97c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0017070 con 0x7f97c815b010 2024-03-31T15:54:46.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c000aca0 con 0x7f97c815b010 2024-03-31T15:54:46.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c80c54f0 con 0x7f97c815b010 2024-03-31T15:54:46.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0012620 con 0x7f97c815b010 2024-03-31T15:54:46.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c80b5850 con 0x7f97c815b010 2024-03-31T15:54:46.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c80686d0 con 0x7f97c815b010 2024-03-31T15:54:46.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f97c001a070 con 0x7f97c815b010 2024-03-31T15:54:46.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 0x7f97a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:46.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f97c0086b60 con 0x7f97c815b010 2024-03-31T15:54:46.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.889+0000 7f97c7400640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 0x7f97a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:46.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.893+0000 7f97c7400640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 0x7f97a406a870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97b80af000 tx=0x7f97b80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:46.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:46.893+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f97c0056d10 con 0x7f97c815b010 2024-03-31T15:54:47.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.057+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97c80cf6d0 con 0x7f97c815b010 2024-03-31T15:54:47.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.057+0000 7f97bf400640 1 -- 172.21.15.90:0/970054490 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f97c0056610 con 0x7f97c815b010 2024-03-31T15:54:47.050 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:47.050 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 msgr2=0x7f97a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 0x7f97a406a870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97b80af000 tx=0x7f97b80a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 msgr2=0x7f97c80c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f97c000cbc0 tx=0x7f97c000a060 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 shutdown_connections 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f97a40683b0 0x7f97a406a870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97c815b010 0x7f97c80c6d60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97c80723c0 0x7f97c8150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 --2- 172.21.15.90:0/970054490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97c806d2a0 0x7f97c80b6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 >> 172.21.15.90:0/970054490 conn(0x7f97c80d13a0 msgr2=0x7f97c806bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 shutdown_connections 2024-03-31T15:54:47.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.061+0000 7f97ce200640 1 -- 172.21.15.90:0/970054490 wait complete. 2024-03-31T15:54:47.071 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:47.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:47.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.217+0000 7f62d5200640 1 Processor -- start 2024-03-31T15:54:47.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.217+0000 7f62d5200640 1 -- start start 2024-03-31T15:54:47.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.217+0000 7f62d5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62d006d020 0x7f62d00ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.217+0000 7f62d5200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d490 0x7f62d00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.217+0000 7f62d5200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d015e8c0 con 0x7f62d015e4e0 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62d5200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f62d00d0ff0 con 0x7f62d006d020 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62d5200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f62d00d12d0 con 0x7f62d006d490 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62d006d020 0x7f62d00ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f62d006d020 0x7f62d00ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37112/0 (socket says 172.21.15.90:37112) 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 -- 172.21.15.90:0/3254937072 learned_addr learned my addr 172.21.15.90:0/3254937072 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62ce800640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d490 0x7f62d00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 -- 172.21.15.90:0/3254937072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d490 msgr2=0x7f62d00cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d490 0x7f62d00cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 -- 172.21.15.90:0/3254937072 >> v1:172.21.15.102:6789/0 conn(0x7f62d015e4e0 legacy=0x7f62d006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 -- 172.21.15.90:0/3254937072 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d00cc710 con 0x7f62d006d020 2024-03-31T15:54:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62c7e00640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d006d020 0x7f62d00ccf10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f62b80098a0 tx=0x7f62b801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e499aa77b6aa29aa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62cd400640 1 -- 172.21.15.90:0/3254937072 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b80b0070 con 0x7f62d006d020 2024-03-31T15:54:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62cd400640 1 -- 172.21.15.90:0/3254937072 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b80ab600 con 0x7f62d006d020 2024-03-31T15:54:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.221+0000 7f62cd400640 1 -- 172.21.15.90:0/3254937072 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b80ab9d0 con 0x7f62d006d020 2024-03-31T15:54:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- 172.21.15.90:0/3254937072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d006d020 msgr2=0x7f62d00ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d006d020 0x7f62d00ccf10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f62b80098a0 tx=0x7f62b801c4b0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- 172.21.15.90:0/3254937072 shutdown_connections 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d490 0x7f62d00cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- 172.21.15.90:0/3254937072 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d006d020 0x7f62d00ccf10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- 172.21.15.90:0/3254937072 >> 172.21.15.90:0/3254937072 conn(0x7f62d00d1f20 msgr2=0x7f62d00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- 172.21.15.90:0/3254937072 shutdown_connections 2024-03-31T15:54:47.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- 172.21.15.90:0/3254937072 wait complete. 2024-03-31T15:54:47.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 Processor -- start 2024-03-31T15:54:47.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- start start 2024-03-31T15:54:47.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d020 0x7f62d01529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62d006d490 0x7f62d0150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00d0d90 con 0x7f62d015e4e0 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00d0c10 con 0x7f62d006d490 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62d5200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f62d016b030 con 0x7f62d006d020 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37114/0 (socket says 172.21.15.90:37114) 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 -- 172.21.15.90:0/1983057088 learned_addr learned my addr 172.21.15.90:0/1983057088 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62c7e00640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62d006d490 0x7f62d0150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62cde00640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d020 0x7f62d01529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 -- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d020 msgr2=0x7f62d01529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d020 0x7f62d01529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 -- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62d006d490 msgr2=0x7f62d0150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62d006d490 0x7f62d0150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.225+0000 7f62ce800640 1 -- 172.21.15.90:0/1983057088 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d0151090 con 0x7f62d015e4e0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62ce800640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f62c8007e40 tx=0x7f62c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8014620 con 0x7f62d015e4e0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c8019070 con 0x7f62d015e4e0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d0151330 con 0x7f62d015e4e0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8014a20 con 0x7f62d015e4e0 2024-03-31T15:54:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d01518f0 con 0x7f62d015e4e0 2024-03-31T15:54:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d00685f0 con 0x7f62d015e4e0 2024-03-31T15:54:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62c8014bc0 con 0x7f62d015e4e0 2024-03-31T15:54:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 0x7f62ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f62c8087600 con 0x7f62d015e4e0 2024-03-31T15:54:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.229+0000 7f62cde00640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 0x7f62ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.233+0000 7f62cde00640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 0x7f62ac06a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62c00015b0 tx=0x7f62c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.233+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f62c80577c0 con 0x7f62d015e4e0 2024-03-31T15:54:47.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.397+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f62d00742b0 con 0x7f62d015e4e0 2024-03-31T15:54:47.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.397+0000 7f62c6000640 1 -- 172.21.15.90:0/1983057088 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f62c80570c0 con 0x7f62d015e4e0 2024-03-31T15:54:47.390 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 msgr2=0x7f62ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 0x7f62ac06a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62c00015b0 tx=0x7f62c0009040 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 msgr2=0x7f62d00c7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f62c8007e40 tx=0x7f62c800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 shutdown_connections 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62ac068380 0x7f62ac06a840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f62d015e4e0 0x7f62d00c7950 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62d006d490 0x7f62d0150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 --2- 172.21.15.90:0/1983057088 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f62d006d020 0x7f62d01529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 >> 172.21.15.90:0/1983057088 conn(0x7f62d00d1f20 msgr2=0x7f62d00c5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 shutdown_connections 2024-03-31T15:54:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.401+0000 7f62d5200640 1 -- 172.21.15.90:0/1983057088 wait complete. 2024-03-31T15:54:47.411 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.557+0000 7f701cc00640 1 Processor -- start 2024-03-31T15:54:47.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.557+0000 7f701cc00640 1 -- start start 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f701cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70180d2b20 0x7f70180cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f701cc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70180cefc0 con 0x7f70180cd640 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f701cc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70180cf2a0 con 0x7f701815c320 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f701cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f70180aed70 con 0x7f70180d2b20 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f70180cd640 0x7f70180af200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44758/0 (socket says 172.21.15.90:44758) 2024-03-31T15:54:47.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 -- 172.21.15.90:0/1309386819 learned_addr learned my addr 172.21.15.90:0/1309386819 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f700fe00640 1 --2- 172.21.15.90:0/1309386819 >> v2:172.21.15.102:3300/0 conn(0x7f70180d2b20 0x7f70180cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238341196 0 0) 0x7f70180cefc0 con 0x7f70180cd640 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ffc003530 con 0x7f70180cd640 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3598787217 0 0) 0x7f6ffc003530 con 0x7f70180cd640 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 >> v2:172.21.15.102:3300/0 conn(0x7f70180d2b20 msgr2=0x7f70180cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:47.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 --2- 172.21.15.90:0/1309386819 >> v2:172.21.15.102:3300/0 conn(0x7f70180d2b20 0x7f70180cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 >> v1:172.21.15.102:6789/0 conn(0x7f701815c320 legacy=0x7f70180cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70180cf580 con 0x7f70180cd640 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7010001a00 con 0x7f70180cd640 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f700fe00640 1 --2- 172.21.15.90:0/1309386819 >> v2:172.21.15.102:3300/0 conn(0x7f70180d2b20 0x7f70180cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 >> v1:172.21.15.90:6789/0 conn(0x7f70180cd640 legacy=0x7f70180af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ffc0039c0 0x7f6ffc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ffc004810 0x7f6ffc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f6ffc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70180aed70 con 0x7f6ffc0051d0 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc00c170 con 0x7f6ffc004810 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc00c370 con 0x7f6ffc0039c0 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f6ffc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7015e00640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ffc0039c0 0x7f6ffc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f700fe00640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ffc004810 0x7f6ffc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 -- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ffc0039c0 msgr2=0x7f6ffc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ffc0039c0 0x7f6ffc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 -- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ffc004810 msgr2=0x7f6ffc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ffc004810 0x7f6ffc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.561+0000 7f7016800640 1 -- 172.21.15.90:0/1309386819 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc003530 con 0x7f6ffc0051d0 2024-03-31T15:54:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.565+0000 7f7016800640 1 -- 172.21.15.90:0/1309386819 reap_dead start 2024-03-31T15:54:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.565+0000 7f7016800640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f6ffc00b9d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f70180cdad0 tx=0x7f701001fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb1e7e22b19713a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.565+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701000a4a0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70100aa4f0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015400640 1 -- 172.21.15.90:0/1309386819 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100aa8c0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 msgr2=0x7f6ffc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f6ffc00b9d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f70180cdad0 tx=0x7f701001fac0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- 172.21.15.90:0/1309386819 shutdown_connections 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 --2- 172.21.15.90:0/1309386819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f6ffc00b9d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- 172.21.15.90:0/1309386819 >> 172.21.15.90:0/1309386819 conn(0x7f70180d1f20 msgr2=0x7f70180d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- 172.21.15.90:0/1309386819 shutdown_connections 2024-03-31T15:54:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- 172.21.15.90:0/1309386819 wait complete. 2024-03-31T15:54:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 Processor -- start 2024-03-31T15:54:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- start start 2024-03-31T15:54:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7018150120 0x7f70181f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70180b1ee0 0x7f70180b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70180ccb70 con 0x7f6ffc0051d0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70180cc9f0 con 0x7f7018150120 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f701cc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f70180cccf0 con 0x7f70180b1ee0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37138/0 (socket says 172.21.15.90:37138) 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 -- 172.21.15.90:0/1486856340 learned_addr learned my addr 172.21.15.90:0/1486856340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7016800640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70180b1ee0 0x7f70180b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f700fe00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7018150120 0x7f70181f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 -- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70180b1ee0 msgr2=0x7f70180b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70180b1ee0 0x7f70180b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 -- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7018150120 msgr2=0x7f70181f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7018150120 0x7f70181f4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.569+0000 7f7015e00640 1 -- 172.21.15.90:0/1486856340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70181507b0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f7015e00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f700800f7f0 tx=0x7f700800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70080050d0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008018070 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7018150a90 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008005490 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7018151050 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70180685f0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f700800d6d0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 0x7f6ff406aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7008086cc0 con 0x7f6ffc0051d0 2024-03-31T15:54:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.573+0000 7f700fe00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 0x7f6ff406aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.577+0000 7f700fe00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 0x7f6ff406aa10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70180b1290 tx=0x7f7000008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.577+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f700808d080 con 0x7f6ffc0051d0 2024-03-31T15:54:47.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:54:47.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:54:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7018150500 con 0x7f6ffc0051d0 2024-03-31T15:54:47.733 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f700e000640 1 -- 172.21.15.90:0/1486856340 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7008056d50 con 0x7f6ffc0051d0 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 msgr2=0x7f6ff406aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 0x7f6ff406aa10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70180b1290 tx=0x7f7000008040 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 msgr2=0x7f701815f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f700800f7f0 tx=0x7f700800fcb0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 shutdown_connections 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ff4068550 0x7f6ff406aa10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70180b1ee0 0x7f70180b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7018150120 0x7f70181f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 --2- 172.21.15.90:0/1486856340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ffc0051d0 0x7f701815f390 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 >> 172.21.15.90:0/1486856340 conn(0x7f70180d1f20 msgr2=0x7f701815abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 shutdown_connections 2024-03-31T15:54:47.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.741+0000 7f701cc00640 1 -- 172.21.15.90:0/1486856340 wait complete. 2024-03-31T15:54:47.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:47.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 Processor -- start 2024-03-31T15:54:47.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 -- start start 2024-03-31T15:54:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff2006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7eff2006ce50 0x7eff2006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff2015e8c0 con 0x7eff200ce7a0 2024-03-31T15:54:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7eff200d0ff0 con 0x7eff2006ce50 2024-03-31T15:54:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff25800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7eff200d12d0 con 0x7eff2015e4e0 2024-03-31T15:54:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff2006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7eff2006ce50 0x7eff2006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7eff200ce7a0 0x7eff200cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:57076/0 (socket says 172.21.15.90:57076) 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1f400640 1 -- 172.21.15.90:0/2811322193 learned_addr learned my addr 172.21.15.90:0/2811322193 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1ea00640 1 -- 172.21.15.90:0/2811322193 >> v2:172.21.15.102:3300/0 conn(0x7eff2006ce50 msgr2=0x7eff2006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1ea00640 1 --2- 172.21.15.90:0/2811322193 >> v2:172.21.15.102:3300/0 conn(0x7eff2006ce50 0x7eff2006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1ea00640 1 -- 172.21.15.90:0/2811322193 >> v1:172.21.15.102:6789/0 conn(0x7eff200ce7a0 legacy=0x7eff200cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1d600640 1 -- 172.21.15.90:0/2811322193 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1448756472 0 0) 0x7eff2015e8c0 con 0x7eff200ce7a0 2024-03-31T15:54:47.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.925+0000 7eff1ea00640 1 -- 172.21.15.90:0/2811322193 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff200cc710 con 0x7eff2015e4e0 2024-03-31T15:54:47.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff1ea00640 1 --2- 172.21.15.90:0/2811322193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff2006c910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff100097c0 tx=0x7eff1001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d7edf73b31cca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff1d600640 1 -- 172.21.15.90:0/2811322193 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff100b0070 con 0x7eff2015e4e0 2024-03-31T15:54:47.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff1d600640 1 -- 172.21.15.90:0/2811322193 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff100ab600 con 0x7eff2015e4e0 2024-03-31T15:54:47.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff1d600640 1 -- 172.21.15.90:0/2811322193 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff100ab9d0 con 0x7eff2015e4e0 2024-03-31T15:54:47.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 -- 172.21.15.90:0/2811322193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 msgr2=0x7eff2006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 --2- 172.21.15.90:0/2811322193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff2006c910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7eff100097c0 tx=0x7eff1001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:47.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 -- 172.21.15.90:0/2811322193 shutdown_connections 2024-03-31T15:54:47.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 --2- 172.21.15.90:0/2811322193 >> v2:172.21.15.102:3300/0 conn(0x7eff2006ce50 0x7eff2006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 --2- 172.21.15.90:0/2811322193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff2006c910 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 -- 172.21.15.90:0/2811322193 >> 172.21.15.90:0/2811322193 conn(0x7eff200d1f20 msgr2=0x7eff200d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:47.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 -- 172.21.15.90:0/2811322193 shutdown_connections 2024-03-31T15:54:47.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 -- 172.21.15.90:0/2811322193 wait complete. 2024-03-31T15:54:47.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.929+0000 7eff25800640 1 Processor -- start 2024-03-31T15:54:47.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- start start 2024-03-31T15:54:47.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff200ce7a0 0x7eff2006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff201542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7eff200d0d90 con 0x7eff2006ce50 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7eff200d0c10 con 0x7eff200ce7a0 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7eff2016b840 con 0x7eff2015e4e0 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37146/0 (socket says 172.21.15.90:37146) 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 -- 172.21.15.90:0/2522427301 learned_addr learned my addr 172.21.15.90:0/2522427301 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1e000640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff200ce7a0 0x7eff2006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1f400640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff201542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 -- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 msgr2=0x7eff201542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff201542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 -- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff200ce7a0 msgr2=0x7eff2006fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff200ce7a0 0x7eff2006fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 -- 172.21.15.90:0/2522427301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff201556e0 con 0x7eff2006ce50 2024-03-31T15:54:47.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff1ea00640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7eff100097c0 tx=0x7eff100191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff100b0040 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff100bc070 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff100b5680 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff201559c0 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff20155f80 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.933+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff200685f0 con 0x7eff2006ce50 2024-03-31T15:54:47.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.937+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7eff100b3070 con 0x7eff2006ce50 2024-03-31T15:54:47.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.937+0000 7eff16a00640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 0x7eff0006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:47.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.937+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7eff10120eb0 con 0x7eff2006ce50 2024-03-31T15:54:47.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.937+0000 7eff1e000640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 0x7eff0006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:47.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.937+0000 7eff1e000640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 0x7eff0006a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff2015afb0 tx=0x7eff0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:47.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:47.941+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7eff10016090 con 0x7eff2006ce50 2024-03-31T15:54:48.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7eff20140d80 con 0x7eff2006ce50 2024-03-31T15:54:48.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff16a00640 1 -- 172.21.15.90:0/2522427301 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7eff100f0980 con 0x7eff2006ce50 2024-03-31T15:54:48.094 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 msgr2=0x7eff0006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 0x7eff0006a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff2015afb0 tx=0x7eff0c009190 comp rx=0 tx=0).stop 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 msgr2=0x7eff20154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7eff100097c0 tx=0x7eff100191b0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 shutdown_connections 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7eff00068380 0x7eff0006a840 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff2015e4e0 0x7eff201542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff200ce7a0 0x7eff2006fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 --2- 172.21.15.90:0/2522427301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff2006ce50 0x7eff20154e90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.101+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 >> 172.21.15.90:0/2522427301 conn(0x7eff200d1f20 msgr2=0x7eff200c6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.105+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 shutdown_connections 2024-03-31T15:54:48.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.105+0000 7eff25800640 1 -- 172.21.15.90:0/2522427301 wait complete. 2024-03-31T15:54:48.114 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.305+0000 7fc4a9c00640 1 Processor -- start 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.305+0000 7fc4a9c00640 1 -- start start 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc4a4070870 0x7fc4a40721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a40ccde0 0x7fc4a415e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a40c70a0 con 0x7fc4a4072730 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a4070c50 con 0x7fc4a4070870 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a4072b90 con 0x7fc4a40ccde0 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc4a4070870 0x7fc4a40721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc4a4070870 0x7fc4a40721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37154/0 (socket says 172.21.15.90:37154) 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 -- 172.21.15.90:0/581408239 learned_addr learned my addr 172.21.15.90:0/581408239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a9200640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a40ccde0 0x7fc4a415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 -- 172.21.15.90:0/581408239 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a40ccde0 msgr2=0x7fc4a415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a40ccde0 0x7fc4a415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 -- 172.21.15.90:0/581408239 >> v1:172.21.15.90:6789/0 conn(0x7fc4a4072730 legacy=0x7fc4a40cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3e00640 1 -- 172.21.15.90:0/581408239 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a40cd320 con 0x7fc4a4070870 2024-03-31T15:54:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.309+0000 7fc4a3400640 1 -- 172.21.15.90:0/581408239 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440305030 0 0) 0x7fc4a40c70a0 con 0x7fc4a4072730 2024-03-31T15:54:48.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4070870 0x7fc4a40721f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47aec2fa264b9a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a3400640 1 -- 172.21.15.90:0/581408239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940b0070 con 0x7fc4a4070870 2024-03-31T15:54:48.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a3400640 1 -- 172.21.15.90:0/581408239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4940ab600 con 0x7fc4a4070870 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a3400640 1 -- 172.21.15.90:0/581408239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940ab9d0 con 0x7fc4a4070870 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- 172.21.15.90:0/581408239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4070870 msgr2=0x7fc4a40721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4070870 0x7fc4a40721f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc494009870 tx=0x7fc49401c560 comp rx=0 tx=0).stop 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- 172.21.15.90:0/581408239 shutdown_connections 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a40ccde0 0x7fc4a415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/581408239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4070870 0x7fc4a40721f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- 172.21.15.90:0/581408239 >> 172.21.15.90:0/581408239 conn(0x7fc4a40d0c60 msgr2=0x7fc4a40d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- 172.21.15.90:0/581408239 shutdown_connections 2024-03-31T15:54:48.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- 172.21.15.90:0/581408239 wait complete. 2024-03-31T15:54:48.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 Processor -- start 2024-03-31T15:54:48.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- start start 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 0x7fc4a414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a40ccde0 0x7fc4a4151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a415c2c0 con 0x7fc4a4072730 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a415c140 con 0x7fc4a40ccde0 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc4a9c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a4167ce0 con 0x7fc4a4070870 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc49be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc49be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37158/0 (socket says 172.21.15.90:37158) 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.313+0000 7fc49be00640 1 -- 172.21.15.90:0/382182027 learned_addr learned my addr 172.21.15.90:0/382182027 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:48.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a9200640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a40ccde0 0x7fc4a4151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 0x7fc4a414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 -- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 msgr2=0x7fc4a414f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 0x7fc4a414f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 -- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a40ccde0 msgr2=0x7fc4a4151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a40ccde0 0x7fc4a4151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 -- 172.21.15.90:0/382182027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a40c8a20 con 0x7fc4a4072730 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 0x7fc4a414f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc49be00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c4d0 tx=0x7fc48c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c017070 con 0x7fc4a4072730 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc48c005480 con 0x7fc4a4072730 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c003840 con 0x7fc4a4072730 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a40c8d00 con 0x7fc4a4072730 2024-03-31T15:54:48.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a40c92c0 con 0x7fc4a4072730 2024-03-31T15:54:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a4068d00 con 0x7fc4a4072730 2024-03-31T15:54:48.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc48c01a070 con 0x7fc4a4072730 2024-03-31T15:54:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 0x7fc48006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.317+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc48c0861e0 con 0x7fc4a4072730 2024-03-31T15:54:48.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.321+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 0x7fc48006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.321+0000 7fc4a3e00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 0x7fc48006a840 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc494008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.321+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc48c056460 con 0x7fc4a4072730 2024-03-31T15:54:48.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.485+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4a414f730 con 0x7fc4a4072730 2024-03-31T15:54:48.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a0c00640 1 -- 172.21.15.90:0/382182027 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fc48c055d60 con 0x7fc4a4072730 2024-03-31T15:54:48.481 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 msgr2=0x7fc48006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 0x7fc48006a840 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc494008040 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 msgr2=0x7fc4a4151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00c4d0 tx=0x7fc48c00c990 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 shutdown_connections 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc480068380 0x7fc48006a840 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a40ccde0 0x7fc4a4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a4072730 0x7fc4a4151680 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 --2- 172.21.15.90:0/382182027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc4a4070870 0x7fc4a414f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 >> 172.21.15.90:0/382182027 conn(0x7fc4a40d0c60 msgr2=0x7fc4a415c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 shutdown_connections 2024-03-31T15:54:48.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.489+0000 7fc4a9c00640 1 -- 172.21.15.90:0/382182027 wait complete. 2024-03-31T15:54:48.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:48.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 Processor -- start 2024-03-31T15:54:48.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 -- start start 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ffb6015d240 0x7ffb600ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb600d0ea0 con 0x7ffb600ce910 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb600cd460 con 0x7ffb60163bd0 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb68400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ffb600cd740 con 0x7ffb6015d240 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffb600ce910 0x7ffb600cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44788/0 (socket says 172.21.15.90:44788) 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65a00640 1 -- 172.21.15.90:0/2407516840 learned_addr learned my addr 172.21.15.90:0/2407516840 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:48.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65000640 1 --2- 172.21.15.90:0/2407516840 >> v2:172.21.15.90:3300/0 conn(0x7ffb6015d240 0x7ffb600ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb5fe00640 1 -- 172.21.15.90:0/2407516840 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133726598 0 0) 0x7ffb600d0ea0 con 0x7ffb600ce910 2024-03-31T15:54:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65000640 1 -- 172.21.15.90:0/2407516840 >> v1:172.21.15.102:6789/0 conn(0x7ffb60163bd0 legacy=0x7ffb60165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65000640 1 -- 172.21.15.90:0/2407516840 >> v1:172.21.15.90:6789/0 conn(0x7ffb600ce910 legacy=0x7ffb600cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.645+0000 7ffb65000640 1 -- 172.21.15.90:0/2407516840 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb600cda20 con 0x7ffb6015d240 2024-03-31T15:54:48.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb65000640 1 --2- 172.21.15.90:0/2407516840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb600ccb10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009740 tx=0x7ffb4c01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71046f08701b2e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb5fe00640 1 -- 172.21.15.90:0/2407516840 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0b0070 con 0x7ffb6015d240 2024-03-31T15:54:48.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb5fe00640 1 -- 172.21.15.90:0/2407516840 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c0ab600 con 0x7ffb6015d240 2024-03-31T15:54:48.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb5fe00640 1 -- 172.21.15.90:0/2407516840 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0ab9d0 con 0x7ffb6015d240 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- 172.21.15.90:0/2407516840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 msgr2=0x7ffb600ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 --2- 172.21.15.90:0/2407516840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb600ccb10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009740 tx=0x7ffb4c01c210 comp rx=0 tx=0).stop 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- 172.21.15.90:0/2407516840 shutdown_connections 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 --2- 172.21.15.90:0/2407516840 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb600ccb10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- 172.21.15.90:0/2407516840 >> 172.21.15.90:0/2407516840 conn(0x7ffb6008bfd0 msgr2=0x7ffb600d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- 172.21.15.90:0/2407516840 shutdown_connections 2024-03-31T15:54:48.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- 172.21.15.90:0/2407516840 wait complete. 2024-03-31T15:54:48.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 Processor -- start 2024-03-31T15:54:48.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.653+0000 7ffb68400640 1 -- start start 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb600ce910 0x7ffb6014a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb60163bd0 0x7ffb60165a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6016a080 con 0x7ffb6015d240 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffb60169f00 con 0x7ffb600ce910 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6016a200 con 0x7ffb60163bd0 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37174/0 (socket says 172.21.15.90:37174) 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb66400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb60163bd0 0x7ffb60165a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 -- 172.21.15.90:0/2767958267 learned_addr learned my addr 172.21.15.90:0/2767958267 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65a00640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb600ce910 0x7ffb6014a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 -- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb60163bd0 msgr2=0x7ffb60165a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb60163bd0 0x7ffb60165a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 -- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb600ce910 msgr2=0x7ffb6014a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb600ce910 0x7ffb6014a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 -- 172.21.15.90:0/2767958267 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb60166530 con 0x7ffb6015d240 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb65000640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c018e20 tx=0x7ffb4c018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0b8020 con 0x7ffb6015d240 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c0b4410 con 0x7ffb6015d240 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0b0040 con 0x7ffb6015d240 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb60166810 con 0x7ffb6015d240 2024-03-31T15:54:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb600c92b0 con 0x7ffb6015d240 2024-03-31T15:54:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.657+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb60068660 con 0x7ffb6015d240 2024-03-31T15:54:48.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.661+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffb4c0bb020 con 0x7ffb6015d240 2024-03-31T15:54:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.661+0000 7ffb5d600640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 0x7ffb4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.661+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffb4c1207c0 con 0x7ffb6015d240 2024-03-31T15:54:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.661+0000 7ffb65a00640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 0x7ffb4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.661+0000 7ffb65a00640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 0x7ffb4006a870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007bc0 tx=0x7ffb54007b10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.665+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffb4c0c2030 con 0x7ffb6015d240 2024-03-31T15:54:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.833+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffb600d0e10 con 0x7ffb6015d240 2024-03-31T15:54:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.833+0000 7ffb5d600640 1 -- 172.21.15.90:0/2767958267 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7ffb4c015f50 con 0x7ffb6015d240 2024-03-31T15:54:48.826 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:48.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 msgr2=0x7ffb4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 0x7ffb4006a870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007bc0 tx=0x7ffb54007b10 comp rx=0 tx=0).stop 2024-03-31T15:54:48.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 msgr2=0x7ffb60165520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:48.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c018e20 tx=0x7ffb4c018fb0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 shutdown_connections 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffb400683b0 0x7ffb4006a870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffb60163bd0 0x7ffb60165a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb6015d240 0x7ffb60165520 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 --2- 172.21.15.90:0/2767958267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb600ce910 0x7ffb6014a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 >> 172.21.15.90:0/2767958267 conn(0x7ffb6008bfd0 msgr2=0x7ffb60151970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 shutdown_connections 2024-03-31T15:54:48.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:48.837+0000 7ffb68400640 1 -- 172.21.15.90:0/2767958267 wait complete. 2024-03-31T15:54:48.848 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:54:48.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:48.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:54:48.868 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:54:48.868 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:54:48.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:48.868 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:54:48.920 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:48.920 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:48.929 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:48.929 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:48.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:49.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 Processor -- start 2024-03-31T15:54:49.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 -- start start 2024-03-31T15:54:49.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c2415f390 0x7f2c240ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c2415fb10 con 0x7f2c24073440 2024-03-31T15:54:49.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c2415fdf0 con 0x7f2c24067330 2024-03-31T15:54:49.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2ac00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2c24160a00 con 0x7f2c2415f390 2024-03-31T15:54:49.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2a200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2c24073440 0x7f2c240d03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44800/0 (socket says 172.21.15.90:44800) 2024-03-31T15:54:49.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c2a200640 1 -- 172.21.15.90:0/1968473396 learned_addr learned my addr 172.21.15.90:0/1968473396 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:49.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c29800640 1 --2- 172.21.15.90:0/1968473396 >> v2:172.21.15.102:3300/0 conn(0x7f2c2415f390 0x7f2c240ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c23e00640 1 -- 172.21.15.90:0/1968473396 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3670394804 0 0) 0x7f2c2415fb10 con 0x7f2c24073440 2024-03-31T15:54:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c23e00640 1 -- 172.21.15.90:0/1968473396 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c0c003530 con 0x7f2c24073440 2024-03-31T15:54:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c29800640 1 -- 172.21.15.90:0/1968473396 >> v1:172.21.15.102:6789/0 conn(0x7f2c24067330 legacy=0x7f2c24072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c29800640 1 -- 172.21.15.90:0/1968473396 >> v1:172.21.15.90:6789/0 conn(0x7f2c24073440 legacy=0x7f2c240d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.125+0000 7f2c29800640 1 -- 172.21.15.90:0/1968473396 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c241600d0 con 0x7f2c2415f390 2024-03-31T15:54:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c29800640 1 --2- 172.21.15.90:0/1968473396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2415f390 0x7f2c240ae9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2c180097c0 tx=0x7f2c1801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1109e6b4fdeceac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c23e00640 1 -- 172.21.15.90:0/1968473396 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180b1070 con 0x7f2c2415f390 2024-03-31T15:54:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c23e00640 1 -- 172.21.15.90:0/1968473396 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c180ac410 con 0x7f2c2415f390 2024-03-31T15:54:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c23e00640 1 -- 172.21.15.90:0/1968473396 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180ac7e0 con 0x7f2c2415f390 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 -- 172.21.15.90:0/1968473396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2415f390 msgr2=0x7f2c240ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/1968473396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2415f390 0x7f2c240ae9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2c180097c0 tx=0x7f2c1801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 -- 172.21.15.90:0/1968473396 shutdown_connections 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/1968473396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c2415f390 0x7f2c240ae9c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 -- 172.21.15.90:0/1968473396 >> 172.21.15.90:0/1968473396 conn(0x7f2c240b5130 msgr2=0x7f2c240b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 -- 172.21.15.90:0/1968473396 shutdown_connections 2024-03-31T15:54:49.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.129+0000 7f2c2ac00640 1 -- 172.21.15.90:0/1968473396 wait complete. 2024-03-31T15:54:49.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 Processor -- start 2024-03-31T15:54:49.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- start start 2024-03-31T15:54:49.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 0x7f2c24099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c2415f390 0x7f2c2415c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2415f8f0 con 0x7f2c24067330 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2415f770 con 0x7f2c24073440 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2c24169e80 con 0x7f2c2415f390 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37182/0 (socket says 172.21.15.90:37182) 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 -- 172.21.15.90:0/655568109 learned_addr learned my addr 172.21.15.90:0/655568109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c28e00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 0x7f2c24099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2a200640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c2415f390 0x7f2c2415c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 -- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c2415f390 msgr2=0x7f2c2415c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c2415f390 0x7f2c2415c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 -- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 msgr2=0x7f2c24099850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 0x7f2c24099850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 -- 172.21.15.90:0/655568109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2406da80 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c28e00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 0x7f2c24099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c29800640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2c180af040 tx=0x7f2c18019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180b9070 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c180acd90 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c240cd510 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180b1040 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.133+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c24156100 con 0x7f2c24067330 2024-03-31T15:54:49.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.137+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c24068d00 con 0x7f2c24067330 2024-03-31T15:54:49.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.137+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2c180bc070 con 0x7f2c24067330 2024-03-31T15:54:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.137+0000 7f2c21600640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 0x7f2c0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.137+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2c18120920 con 0x7f2c24067330 2024-03-31T15:54:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.137+0000 7f2c28e00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 0x7f2c0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.141+0000 7f2c28e00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 0x7f2c0406a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2c2415e980 tx=0x7f2c10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.141+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2c18016090 con 0x7f2c24067330 2024-03-31T15:54:49.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.301+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c240724c0 con 0x7f2c24067330 2024-03-31T15:54:49.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.301+0000 7f2c21600640 1 -- 172.21.15.90:0/655568109 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f2c180f0a40 con 0x7f2c24067330 2024-03-31T15:54:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 msgr2=0x7f2c0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 0x7f2c0406a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2c2415e980 tx=0x7f2c10009040 comp rx=0 tx=0).stop 2024-03-31T15:54:49.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 msgr2=0x7f2c241568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2c180af040 tx=0x7f2c18019180 comp rx=0 tx=0).stop 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 shutdown_connections 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2c04068380 0x7f2c0406a840 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c2415f390 0x7f2c2415c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c24073440 0x7f2c24099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 --2- 172.21.15.90:0/655568109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c24067330 0x7f2c241568d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 >> 172.21.15.90:0/655568109 conn(0x7f2c240b5130 msgr2=0x7f2c240cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 shutdown_connections 2024-03-31T15:54:49.297 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.305+0000 7f2c2ac00640 1 -- 172.21.15.90:0/655568109 wait complete. 2024-03-31T15:54:49.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:49.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:49.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 Processor -- start 2024-03-31T15:54:49.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 -- start start 2024-03-31T15:54:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe6900b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6900d0a30 0x7fe6900ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6900cce10 con 0x7fe6900cc850 2024-03-31T15:54:49.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe69015c520 con 0x7fe6900d0a30 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68f200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe6900cd0f0 con 0x7fe69015c140 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6900d0a30 0x7fe6900ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6900d0a30 0x7fe6900ae9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37186/0 (socket says 172.21.15.90:37186) 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68d400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe6900cc850 0x7fe6900d0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44806/0 (socket says 172.21.15.90:44806) 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.461+0000 7fe68e800640 1 -- 172.21.15.90:0/3806856486 learned_addr learned my addr 172.21.15.90:0/3806856486 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68de00640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe6900b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68e800640 1 -- 172.21.15.90:0/3806856486 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 msgr2=0x7fe6900b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68e800640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe6900b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68e800640 1 -- 172.21.15.90:0/3806856486 >> v1:172.21.15.90:6789/0 conn(0x7fe6900cc850 legacy=0x7fe6900d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68e800640 1 -- 172.21.15.90:0/3806856486 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6900cd3d0 con 0x7fe6900d0a30 2024-03-31T15:54:49.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68ca00640 1 -- 172.21.15.90:0/3806856486 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458504757 0 0) 0x7fe6900cce10 con 0x7fe6900cc850 2024-03-31T15:54:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68e800640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900d0a30 0x7fe6900ae9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe688009e50 tx=0x7fe68801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb84d85dcf2bc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68ca00640 1 -- 172.21.15.90:0/3806856486 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6880a58d0 con 0x7fe6900d0a30 2024-03-31T15:54:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68ca00640 1 -- 172.21.15.90:0/3806856486 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6880a5a70 con 0x7fe6900d0a30 2024-03-31T15:54:49.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68ca00640 1 -- 172.21.15.90:0/3806856486 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6880a5e40 con 0x7fe6900d0a30 2024-03-31T15:54:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 -- 172.21.15.90:0/3806856486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900d0a30 msgr2=0x7fe6900ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900d0a30 0x7fe6900ae9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe688009e50 tx=0x7fe68801ca00 comp rx=0 tx=0).stop 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 -- 172.21.15.90:0/3806856486 shutdown_connections 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900d0a30 0x7fe6900ae9c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 --2- 172.21.15.90:0/3806856486 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe6900b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.465+0000 7fe68f200640 1 -- 172.21.15.90:0/3806856486 >> 172.21.15.90:0/3806856486 conn(0x7fe6900d1d40 msgr2=0x7fe6900d2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- 172.21.15.90:0/3806856486 shutdown_connections 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- 172.21.15.90:0/3806856486 wait complete. 2024-03-31T15:54:49.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 Processor -- start 2024-03-31T15:54:49.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- start start 2024-03-31T15:54:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6900d0a30 0x7fe690156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe69015de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6900af890 con 0x7fe6900cc850 2024-03-31T15:54:49.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6900af710 con 0x7fe6900d0a30 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe6900afa10 con 0x7fe69015c140 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37200/0 (socket says 172.21.15.90:37200) 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 -- 172.21.15.90:0/1162749862 learned_addr learned my addr 172.21.15.90:0/1162749862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68e800640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe69015de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68d400640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6900d0a30 0x7fe690156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 -- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 msgr2=0x7fe69015de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe69015de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 -- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6900d0a30 msgr2=0x7fe690156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6900d0a30 0x7fe690156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 -- 172.21.15.90:0/1162749862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe690156b60 con 0x7fe6900cc850 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68de00640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe68000d830 tx=0x7fe68000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000b6c0 con 0x7fe6900cc850 2024-03-31T15:54:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe680019070 con 0x7fe6900cc850 2024-03-31T15:54:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000ba60 con 0x7fe6900cc850 2024-03-31T15:54:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6901576d0 con 0x7fe6900cc850 2024-03-31T15:54:49.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.469+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe690157c90 con 0x7fe6900cc850 2024-03-31T15:54:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe690068d00 con 0x7fe6900cc850 2024-03-31T15:54:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe68001e460 con 0x7fe6900cc850 2024-03-31T15:54:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe686000640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 0x7fe67006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe6800873f0 con 0x7fe6900cc850 2024-03-31T15:54:49.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe68d400640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 0x7fe67006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.468 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.473+0000 7fe68d400640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 0x7fe67006a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6901587c0 tx=0x7fe67c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.477+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe690156b60 con 0x7fe6900cc850 2024-03-31T15:54:49.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe69006f710 con 0x7fe6900cc850 2024-03-31T15:54:49.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe686000640 1 -- 172.21.15.90:0/1162749862 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fe680057640 con 0x7fe6900cc850 2024-03-31T15:54:49.629 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 msgr2=0x7fe67006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 0x7fe67006a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe6901587c0 tx=0x7fe67c007c10 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 msgr2=0x7fe690156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe68000d830 tx=0x7fe68000dcf0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 shutdown_connections 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe670068380 0x7fe67006a840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe69015c140 0x7fe69015de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6900d0a30 0x7fe690156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 --2- 172.21.15.90:0/1162749862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6900cc850 0x7fe690156620 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 >> 172.21.15.90:0/1162749862 conn(0x7fe6900d1d40 msgr2=0x7fe69015ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 shutdown_connections 2024-03-31T15:54:49.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.637+0000 7fe68f200640 1 -- 172.21.15.90:0/1162749862 wait complete. 2024-03-31T15:54:49.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:49.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 Processor -- start 2024-03-31T15:54:49.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 -- start start 2024-03-31T15:54:49.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb815e4e0 0x7f4fb806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4fb80cdf20 0x7f4fb806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb815e8c0 con 0x7f4fb806ce50 2024-03-31T15:54:49.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb80d0ff0 con 0x7f4fb80cdf20 2024-03-31T15:54:49.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fbd800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb80d12d0 con 0x7f4fb815e4e0 2024-03-31T15:54:49.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4fb80cdf20 0x7f4fb806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4fb80cdf20 0x7f4fb806d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37202/0 (socket says 172.21.15.90:37202) 2024-03-31T15:54:49.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 -- 172.21.15.90:0/2720701440 learned_addr learned my addr 172.21.15.90:0/2720701440 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:49.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb6a00640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb815e4e0 0x7f4fb806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 -- 172.21.15.90:0/2720701440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb815e4e0 msgr2=0x7f4fb806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:49.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb815e4e0 0x7f4fb806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 -- 172.21.15.90:0/2720701440 >> v1:172.21.15.102:6789/0 conn(0x7f4fb806ce50 legacy=0x7f4fb80ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.797+0000 7f4fb7400640 1 -- 172.21.15.90:0/2720701440 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb80cc710 con 0x7f4fb80cdf20 2024-03-31T15:54:49.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.801+0000 7f4fb7400640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb80cdf20 0x7f4fb806d290 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009e50 tx=0x7f4fb001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628b818fdfa18f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.801+0000 7f4fb5600640 1 -- 172.21.15.90:0/2720701440 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00a5660 con 0x7f4fb80cdf20 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.801+0000 7f4fb5600640 1 -- 172.21.15.90:0/2720701440 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb00a5800 con 0x7f4fb80cdf20 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.801+0000 7f4fb5600640 1 -- 172.21.15.90:0/2720701440 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00a5bd0 con 0x7f4fb80cdf20 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- 172.21.15.90:0/2720701440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb80cdf20 msgr2=0x7f4fb806d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb80cdf20 0x7f4fb806d290 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009e50 tx=0x7f4fb001c9a0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- 172.21.15.90:0/2720701440 shutdown_connections 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb80cdf20 0x7f4fb806d290 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- 172.21.15.90:0/2720701440 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb815e4e0 0x7f4fb806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- 172.21.15.90:0/2720701440 >> 172.21.15.90:0/2720701440 conn(0x7f4fb80d1f20 msgr2=0x7f4fb80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- 172.21.15.90:0/2720701440 shutdown_connections 2024-03-31T15:54:49.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- 172.21.15.90:0/2720701440 wait complete. 2024-03-31T15:54:49.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 Processor -- start 2024-03-31T15:54:49.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- start start 2024-03-31T15:54:49.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fb806ce50 0x7f4fb8157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb80cdf20 0x7f4fb8155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb816b1b0 con 0x7f4fb815e4e0 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb816b030 con 0x7f4fb806ce50 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fbd800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb816b330 con 0x7f4fb80cdf20 2024-03-31T15:54:49.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37204/0 (socket says 172.21.15.90:37204) 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 -- 172.21.15.90:0/4033948414 learned_addr learned my addr 172.21.15.90:0/4033948414 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb6a00640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fb806ce50 0x7f4fb8157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb6000640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb80cdf20 0x7f4fb8155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 -- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb80cdf20 msgr2=0x7f4fb8155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb80cdf20 0x7f4fb8155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 -- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fb806ce50 msgr2=0x7f4fb8157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fb806ce50 0x7f4fb8157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 -- 172.21.15.90:0/4033948414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb815a920 con 0x7f4fb815e4e0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.805+0000 7f4fb7400640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00af040 tx=0x7f4fb0019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00bc020 con 0x7f4fb815e4e0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb00b8670 con 0x7f4fb815e4e0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb815ac00 con 0x7f4fb815e4e0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0009510 con 0x7f4fb815e4e0 2024-03-31T15:54:49.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb8156620 con 0x7f4fb815e4e0 2024-03-31T15:54:49.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb80685f0 con 0x7f4fb815e4e0 2024-03-31T15:54:49.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.809+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4fb00bf020 con 0x7f4fb815e4e0 2024-03-31T15:54:49.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.813+0000 7f4faea00640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 0x7f4f9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:49.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.813+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4fb0121aa0 con 0x7f4fb815e4e0 2024-03-31T15:54:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.813+0000 7f4fb6a00640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 0x7f4f9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:49.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.813+0000 7f4fb6a00640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 0x7f4f9806a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4fb81fa620 tx=0x7f4fa8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:49.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.817+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4fb00c6030 con 0x7f4fb815e4e0 2024-03-31T15:54:49.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4fb81584c0 con 0x7f4fb815e4e0 2024-03-31T15:54:49.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4faea00640 1 -- 172.21.15.90:0/4033948414 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f4fb0016620 con 0x7f4fb815e4e0 2024-03-31T15:54:49.970 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 msgr2=0x7f4f9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 0x7f4f9806a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4fb81fa620 tx=0x7f4fa8009190 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 msgr2=0x7f4fb80c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00af040 tx=0x7f4fb0019740 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 shutdown_connections 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4f98068380 0x7f4f9806a840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4fb815e4e0 0x7f4fb80c6100 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4fb80cdf20 0x7f4fb8155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 --2- 172.21.15.90:0/4033948414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fb806ce50 0x7f4fb8157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 >> 172.21.15.90:0/4033948414 conn(0x7f4fb80d1f20 msgr2=0x7f4fb806d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.977+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 shutdown_connections 2024-03-31T15:54:49.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:49.981+0000 7f4fbd800640 1 -- 172.21.15.90:0/4033948414 wait complete. 2024-03-31T15:54:49.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 Processor -- start 2024-03-31T15:54:50.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 -- start start 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe5240b3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe5240ce620 0x7fe5240cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5240cd300 con 0x7fe52415f670 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe5240b35e0 con 0x7fe5240ce620 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe52a200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe5240b4410 con 0x7fe5240b3a70 2024-03-31T15:54:50.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe529800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe5240ce620 0x7fe5240cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe528e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe52415f670 0x7fe5240d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:57108/0 (socket says 172.21.15.90:57108) 2024-03-31T15:54:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe528e00640 1 -- 172.21.15.90:0/3934873878 learned_addr learned my addr 172.21.15.90:0/3934873878 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe51be00640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe5240b3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe529800640 1 -- 172.21.15.90:0/3934873878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 msgr2=0x7fe5240b3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe529800640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe5240b3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe529800640 1 -- 172.21.15.90:0/3934873878 >> v1:172.21.15.102:6789/0 conn(0x7fe52415f670 legacy=0x7fe5240d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe529800640 1 -- 172.21.15.90:0/3934873878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5240cd580 con 0x7fe5240ce620 2024-03-31T15:54:50.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.141+0000 7fe523e00640 1 -- 172.21.15.90:0/3934873878 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403347238 0 0) 0x7fe514001120 con 0x7fe52415f670 2024-03-31T15:54:50.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe529800640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5240ce620 0x7fe5240cea80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009d90 tx=0x7fe51c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4672af0f10770e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe523e00640 1 -- 172.21.15.90:0/3934873878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b2070 con 0x7fe5240ce620 2024-03-31T15:54:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe523e00640 1 -- 172.21.15.90:0/3934873878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c0a5940 con 0x7fe5240ce620 2024-03-31T15:54:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe523e00640 1 -- 172.21.15.90:0/3934873878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0a5d10 con 0x7fe5240ce620 2024-03-31T15:54:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- 172.21.15.90:0/3934873878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5240ce620 msgr2=0x7fe5240cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5240ce620 0x7fe5240cea80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009d90 tx=0x7fe51c01c970 comp rx=0 tx=0).stop 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- 172.21.15.90:0/3934873878 shutdown_connections 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5240ce620 0x7fe5240cea80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 --2- 172.21.15.90:0/3934873878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe5240b3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- 172.21.15.90:0/3934873878 >> 172.21.15.90:0/3934873878 conn(0x7fe5240d1f80 msgr2=0x7fe5240d2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- 172.21.15.90:0/3934873878 shutdown_connections 2024-03-31T15:54:50.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- 172.21.15.90:0/3934873878 wait complete. 2024-03-31T15:54:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 Processor -- start 2024-03-31T15:54:50.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.149+0000 7fe52a200640 1 -- start start 2024-03-31T15:54:50.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe52415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5240ce620 0x7fe524159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 0x7fe5241599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe524168f20 con 0x7fe52415f670 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe5240d0c00 con 0x7fe5240ce620 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe5241690a0 con 0x7fe5240b3a70 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe528e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe52415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 0x7fe5241599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe528e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe52415dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:60920/0 (socket says 172.21.15.90:60920) 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe528e00640 1 -- 172.21.15.90:0/678059374 learned_addr learned my addr 172.21.15.90:0/678059374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe51be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5240ce620 0x7fe524159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 -- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 msgr2=0x7fe52415dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe52415dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 -- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5240ce620 msgr2=0x7fe524159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5240ce620 0x7fe524159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 -- 172.21.15.90:0/678059374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe524157d80 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe529800640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 0x7fe5241599b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe5240730d0 tx=0x7fe51c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b2040 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c0b8470 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe524158060 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b8820 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe524158620 con 0x7fe52415f670 2024-03-31T15:54:50.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5240685f0 con 0x7fe52415f670 2024-03-31T15:54:50.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.153+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe51c0c1020 con 0x7fe52415f670 2024-03-31T15:54:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.157+0000 7fe521600640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 0x7fe50006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.157+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe51c1214f0 con 0x7fe52415f670 2024-03-31T15:54:50.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.157+0000 7fe528e00640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 0x7fe50006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.157+0000 7fe528e00640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 0x7fe50006a840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f70 tx=0x7fe514007eb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.161+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe51c016560 con 0x7fe52415f670 2024-03-31T15:54:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.373+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe52415fca0 con 0x7fe52415f670 2024-03-31T15:54:50.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.373+0000 7fe521600640 1 -- 172.21.15.90:0/678059374 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fe51c0f0fe0 con 0x7fe52415f670 2024-03-31T15:54:50.366 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 msgr2=0x7fe50006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 0x7fe50006a840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f70 tx=0x7fe514007eb0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 msgr2=0x7fe5241599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 0x7fe5241599b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe5240730d0 tx=0x7fe51c019710 comp rx=0 tx=0).stop 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 shutdown_connections 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe500068380 0x7fe50006a840 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe52415f670 0x7fe5241599b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5240ce620 0x7fe524159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 --2- 172.21.15.90:0/678059374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe5240b3a70 0x7fe52415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 >> 172.21.15.90:0/678059374 conn(0x7fe5240d1f80 msgr2=0x7fe524166450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 shutdown_connections 2024-03-31T15:54:50.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.377+0000 7fe52a200640 1 -- 172.21.15.90:0/678059374 wait complete. 2024-03-31T15:54:50.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:50.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.541+0000 7f385fe00640 1 Processor -- start 2024-03-31T15:54:50.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.541+0000 7f385fe00640 1 -- start start 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f386006d4f0 0x7f38600cab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38600cb580 con 0x7f386006eac0 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38600cb860 con 0x7f38600cb0c0 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f38600cbb40 con 0x7f386006d4f0 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f386006eac0 0x7f386006eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44810/0 (socket says 172.21.15.90:44810) 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385ea00640 1 -- 172.21.15.90:0/1585380777 learned_addr learned my addr 172.21.15.90:0/1585380777 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:50.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f3857e00640 1 --2- 172.21.15.90:0/1585380777 >> v2:172.21.15.90:3300/0 conn(0x7f386006d4f0 0x7f38600cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385e000640 1 -- 172.21.15.90:0/1585380777 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341637400 0 0) 0x7f38600cb580 con 0x7f386006eac0 2024-03-31T15:54:50.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f385e000640 1 -- 172.21.15.90:0/1585380777 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3844003530 con 0x7f386006eac0 2024-03-31T15:54:50.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f3857e00640 1 -- 172.21.15.90:0/1585380777 >> v1:172.21.15.102:6789/0 conn(0x7f38600cb0c0 legacy=0x7f38600c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f3857e00640 1 -- 172.21.15.90:0/1585380777 >> v1:172.21.15.90:6789/0 conn(0x7f386006eac0 legacy=0x7f386006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.545+0000 7f3857e00640 1 -- 172.21.15.90:0/1585380777 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38600cbe20 con 0x7f386006d4f0 2024-03-31T15:54:50.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f3857e00640 1 --2- 172.21.15.90:0/1585380777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f386006d4f0 0x7f38600cab80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f38480097c0 tx=0x7f384801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c1462e58cb84cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385e000640 1 -- 172.21.15.90:0/1585380777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38480b1070 con 0x7f386006d4f0 2024-03-31T15:54:50.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385e000640 1 -- 172.21.15.90:0/1585380777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38480ac410 con 0x7f386006d4f0 2024-03-31T15:54:50.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385e000640 1 -- 172.21.15.90:0/1585380777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38480ac7c0 con 0x7f386006d4f0 2024-03-31T15:54:50.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- 172.21.15.90:0/1585380777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f386006d4f0 msgr2=0x7f38600cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 --2- 172.21.15.90:0/1585380777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f386006d4f0 0x7f38600cab80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f38480097c0 tx=0x7f384801c490 comp rx=0 tx=0).stop 2024-03-31T15:54:50.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- 172.21.15.90:0/1585380777 shutdown_connections 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 --2- 172.21.15.90:0/1585380777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f386006d4f0 0x7f38600cab80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- 172.21.15.90:0/1585380777 >> 172.21.15.90:0/1585380777 conn(0x7f38600cfb50 msgr2=0x7f38600cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- 172.21.15.90:0/1585380777 shutdown_connections 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- 172.21.15.90:0/1585380777 wait complete. 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 Processor -- start 2024-03-31T15:54:50.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- start start 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f386006eac0 0x7f3860143890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 0x7f38600c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f38600c61c0 con 0x7f38600cb0c0 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38600c6040 con 0x7f386006eac0 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.549+0000 7f385fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3860168ca0 con 0x7f386006d4f0 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 0x7f38600c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:60922/0 (socket says 172.21.15.90:60922) 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 -- 172.21.15.90:0/2113183905 learned_addr learned my addr 172.21.15.90:0/2113183905 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:50.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3857e00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f386006eac0 0x7f3860143890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 -- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f386006eac0 msgr2=0x7f3860143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f386006eac0 0x7f3860143890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 -- 172.21.15.90:0/2113183905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 msgr2=0x7f38600c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 0x7f38600c6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 -- 172.21.15.90:0/2113183905 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386013d590 con 0x7f386006d4f0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385f400640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 0x7f38600c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385ea00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f385000c4d0 tx=0x7f385000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850017070 con 0x7f386006d4f0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3850005cf0 con 0x7f386006d4f0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386013d870 con 0x7f386006d4f0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850007270 con 0x7f386006d4f0 2024-03-31T15:54:50.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f386013de30 con 0x7f386006d4f0 2024-03-31T15:54:50.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38600685f0 con 0x7f386006d4f0 2024-03-31T15:54:50.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f385001a070 con 0x7f386006d4f0 2024-03-31T15:54:50.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.553+0000 7f3856a00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 0x7f383c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.557+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f38500861d0 con 0x7f386006d4f0 2024-03-31T15:54:50.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.557+0000 7f3857e00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 0x7f383c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.557+0000 7f3857e00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 0x7f383c06a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f38480ae000 tx=0x7f3848008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.561+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3850055c10 con 0x7f386006d4f0 2024-03-31T15:54:50.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f386006d260 con 0x7f386006d4f0 2024-03-31T15:54:50.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f3856a00640 1 -- 172.21.15.90:0/2113183905 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3850059f10 con 0x7f386006d4f0 2024-03-31T15:54:50.709 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:50.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 msgr2=0x7f383c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 0x7f383c06a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f38480ae000 tx=0x7f3848008040 comp rx=0 tx=0).stop 2024-03-31T15:54:50.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 msgr2=0x7f386013b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f385000c4d0 tx=0x7f385000c990 comp rx=0 tx=0).stop 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 shutdown_connections 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f383c0683b0 0x7f383c06a870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38600cb0c0 0x7f38600c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f386006eac0 0x7f3860143890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 --2- 172.21.15.90:0/2113183905 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f386006d4f0 0x7f386013b020 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 >> 172.21.15.90:0/2113183905 conn(0x7f38600cfb50 msgr2=0x7f38600aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 shutdown_connections 2024-03-31T15:54:50.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.717+0000 7f385fe00640 1 -- 172.21.15.90:0/2113183905 wait complete. 2024-03-31T15:54:50.730 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 Processor -- start 2024-03-31T15:54:50.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 -- start start 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f18401624a0 0x7f18400d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18400b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18400d1400 con 0x7f184015e320 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f184015e700 con 0x7f18401624a0 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.877+0000 7f1848000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f18400b1a20 con 0x7f18400d2940 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f18401624a0 0x7f18400d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1845600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f184015e320 0x7f1840161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:57130/0 (socket says 172.21.15.90:57130) 2024-03-31T15:54:50.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1845600640 1 -- 172.21.15.90:0/202317882 learned_addr learned my addr 172.21.15.90:0/202317882 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:50.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1846000640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18400b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f183fe00640 1 -- 172.21.15.90:0/202317882 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1561435585 0 0) 0x7f18400d1400 con 0x7f184015e320 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f183fe00640 1 -- 172.21.15.90:0/202317882 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1828003530 con 0x7f184015e320 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 -- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 msgr2=0x7f18400b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18400b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 -- 172.21.15.90:0/202317882 >> v1:172.21.15.102:6789/0 conn(0x7f184015e320 legacy=0x7f1840161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 -- 172.21.15.90:0/202317882 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18400b1d00 con 0x7f18401624a0 2024-03-31T15:54:50.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1846000640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18400b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:50.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.881+0000 7f1844c00640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f18400d0d20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f182c0097c0 tx=0x7f182c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ff24f3cba9887a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f183fe00640 1 -- 172.21.15.90:0/202317882 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c0aa620 con 0x7f18401624a0 2024-03-31T15:54:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f183fe00640 1 -- 172.21.15.90:0/202317882 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c0af070 con 0x7f18401624a0 2024-03-31T15:54:50.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f183fe00640 1 -- 172.21.15.90:0/202317882 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c0aa9e0 con 0x7f18401624a0 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 msgr2=0x7f18400d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f18400d0d20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f182c0097c0 tx=0x7f182c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- 172.21.15.90:0/202317882 shutdown_connections 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18400b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 --2- 172.21.15.90:0/202317882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f18400d0d20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- 172.21.15.90:0/202317882 >> 172.21.15.90:0/202317882 conn(0x7f184008bfd0 msgr2=0x7f18400d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- 172.21.15.90:0/202317882 shutdown_connections 2024-03-31T15:54:50.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- 172.21.15.90:0/202317882 wait complete. 2024-03-31T15:54:50.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 Processor -- start 2024-03-31T15:54:50.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.885+0000 7f1848000640 1 -- start start 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 0x7f1840154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f184006d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18400b3620 con 0x7f184015e320 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18400b34a0 con 0x7f18401624a0 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f18400b37a0 con 0x7f18400d2940 2024-03-31T15:54:50.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:60942/0 (socket says 172.21.15.90:60942) 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1846000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f184006d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 -- 172.21.15.90:0/3990826699 learned_addr learned my addr 172.21.15.90:0/3990826699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1844c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 0x7f1840154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 -- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 msgr2=0x7f184006d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f184006d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 -- 172.21.15.90:0/3990826699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 msgr2=0x7f1840154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 0x7f1840154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 -- 172.21.15.90:0/3990826699 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184015b580 con 0x7f18400d2940 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1846000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f184006d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:50.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1844c00640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 0x7f1840154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1845600640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f183400c420 tx=0x7f183400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834017070 con 0x7f18400d2940 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1834005cf0 con 0x7f18400d2940 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18340072d0 con 0x7f18400d2940 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184015b860 con 0x7f18400d2940 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f184015be20 con 0x7f18400d2940 2024-03-31T15:54:50.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.889+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1840068660 con 0x7f18400d2940 2024-03-31T15:54:50.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.893+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1834007470 con 0x7f18400d2940 2024-03-31T15:54:50.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.893+0000 7f183d600640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 0x7f182006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:50.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.893+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1834085ff0 con 0x7f18400d2940 2024-03-31T15:54:50.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.893+0000 7f1844c00640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 0x7f182006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:50.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.893+0000 7f1844c00640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 0x7f182006a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f182c0ae000 tx=0x7f182c0b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:50.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:50.897+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1834055c30 con 0x7f18400d2940 2024-03-31T15:54:51.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.065+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1840068da0 con 0x7f18400d2940 2024-03-31T15:54:51.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f183d600640 1 -- 172.21.15.90:0/3990826699 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1834059f30 con 0x7f18400d2940 2024-03-31T15:54:51.060 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 msgr2=0x7f182006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 0x7f182006a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f182c0ae000 tx=0x7f182c0b5040 comp rx=0 tx=0).stop 2024-03-31T15:54:51.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 msgr2=0x7f18401592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f183400c420 tx=0x7f183400c820 comp rx=0 tx=0).stop 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 shutdown_connections 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1820068380 0x7f182006a840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18401624a0 0x7f184006d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f184015e320 0x7f1840154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 --2- 172.21.15.90:0/3990826699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18400d2940 0x7f18401592f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 >> 172.21.15.90:0/3990826699 conn(0x7f184008bfd0 msgr2=0x7f184015ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 shutdown_connections 2024-03-31T15:54:51.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.069+0000 7f1848000640 1 -- 172.21.15.90:0/3990826699 wait complete. 2024-03-31T15:54:51.082 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:51.082 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:51.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:51.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 Processor -- start 2024-03-31T15:54:51.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 -- start start 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ca40cc9d0 0x7f1ca4142030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca406b820 0x7f1ca406bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca40b34f0 con 0x7f1ca40ccdb0 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca40b3ab0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1caa800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca4141ba0 con 0x7f1ca406b820 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1ca40ccdb0 0x7f1ca40b2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44820/0 (socket says 172.21.15.90:44820) 2024-03-31T15:54:51.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca2a00640 1 -- 172.21.15.90:0/2655284123 learned_addr learned my addr 172.21.15.90:0/2655284123 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2655284123 >> v2:172.21.15.102:3300/0 conn(0x7f1ca40cc9d0 0x7f1ca4142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3e00640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca406b820 0x7f1ca406bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225550056 0 0) 0x7f1ca40b34f0 con 0x7f1ca40ccdb0 2024-03-31T15:54:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c88003530 con 0x7f1ca40ccdb0 2024-03-31T15:54:51.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2453860857 0 0) 0x7f1c88003530 con 0x7f1ca40ccdb0 2024-03-31T15:54:51.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3400640 1 -- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca406b820 msgr2=0x7f1ca406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:51.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca406b820 0x7f1ca406bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3400640 1 -- 172.21.15.90:0/2655284123 >> v1:172.21.15.90:6789/0 conn(0x7f1ca40ccdb0 legacy=0x7f1ca40b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.237+0000 7f1ca3400640 1 -- 172.21.15.90:0/2655284123 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca40b37d0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40cc9d0 0x7f1ca4142030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097c0 tx=0x7f1c9401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab70393e7bbe449c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c940b0070 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c940ab400 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1ca2000640 1 -- 172.21.15.90:0/2655284123 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90001680 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 -- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40cc9d0 msgr2=0x7f1ca4142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40cc9d0 0x7f1ca4142030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097c0 tx=0x7f1c9401c330 comp rx=0 tx=0).stop 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 -- 172.21.15.90:0/2655284123 shutdown_connections 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca406b820 0x7f1ca406bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 --2- 172.21.15.90:0/2655284123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40cc9d0 0x7f1ca4142030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 -- 172.21.15.90:0/2655284123 >> 172.21.15.90:0/2655284123 conn(0x7f1ca40b1c80 msgr2=0x7f1ca40b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 -- 172.21.15.90:0/2655284123 shutdown_connections 2024-03-31T15:54:51.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.241+0000 7f1caa800640 1 -- 172.21.15.90:0/2655284123 wait complete. 2024-03-31T15:54:51.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 Processor -- start 2024-03-31T15:54:51.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- start start 2024-03-31T15:54:51.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 0x7f1ca415df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 0x7f1ca413cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca40cf0c0 con 0x7f1ca406b820 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca40cef40 con 0x7f1ca40ccdb0 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca415c620 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 0x7f1ca415df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37272/0 (socket says 172.21.15.90:37272) 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3400640 1 -- 172.21.15.90:0/2076858705 learned_addr learned my addr 172.21.15.90:0/2076858705 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3e00640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 0x7f1ca413cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 -- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 msgr2=0x7f1ca413cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 0x7f1ca413cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 -- 172.21.15.90:0/2076858705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 msgr2=0x7f1ca414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 -- 172.21.15.90:0/2076858705 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca415ce80 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca3e00640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 0x7f1ca413cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1ca2a00640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 0x7f1ca415df20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000c530 tx=0x7f1c9000c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90005cf0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c90007070 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca415d160 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c90007460 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca415d720 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.245+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca40686d0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.249+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c90007600 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.249+0000 7f1c9b400640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 0x7f1c8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.249+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1c90086bf0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.249+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 0x7f1c8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.249+0000 7f1ca3400640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 0x7f1c8406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097c0 tx=0x7f1c940a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.253+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1c90056740 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ca40b4aa0 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1c9b400640 1 -- 172.21.15.90:0/2076858705 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f1c9005aa40 con 0x7f1ca40cc9d0 2024-03-31T15:54:51.406 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:51.406 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 msgr2=0x7f1c8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 0x7f1c8406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097c0 tx=0x7f1c940a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 msgr2=0x7f1ca415df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 0x7f1ca415df20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c9000c530 tx=0x7f1c9000c9f0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 shutdown_connections 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1c84068380 0x7f1c8406a840 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ca40ccdb0 0x7f1ca413cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ca40cc9d0 0x7f1ca415df20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 --2- 172.21.15.90:0/2076858705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ca406b820 0x7f1ca414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.413+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 >> 172.21.15.90:0/2076858705 conn(0x7f1ca40b1c80 msgr2=0x7f1ca41600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.417+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 shutdown_connections 2024-03-31T15:54:51.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.417+0000 7f1caa800640 1 -- 172.21.15.90:0/2076858705 wait complete. 2024-03-31T15:54:51.427 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:51.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.573+0000 7f6f77c00640 1 Processor -- start 2024-03-31T15:54:51.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.573+0000 7f6f77c00640 1 -- start start 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 0x7f6f7006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f77c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7015e8c0 con 0x7f6f7006ce50 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f77c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f700d0ff0 con 0x7f6f700cdf20 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f77c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6f700d12d0 con 0x7f6f7015e4e0 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f6fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6f7006ce50 0x7f6f700ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44834/0 (socket says 172.21.15.90:44834) 2024-03-31T15:54:51.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f6fe00640 1 -- 172.21.15.90:0/600674287 learned_addr learned my addr 172.21.15.90:0/600674287 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f75200640 1 --2- 172.21.15.90:0/600674287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 0x7f6f7006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f6f400640 1 -- 172.21.15.90:0/600674287 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1080651731 0 0) 0x7f6f7015e8c0 con 0x7f6f7006ce50 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f6f400640 1 -- 172.21.15.90:0/600674287 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f58003530 con 0x7f6f7006ce50 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f75200640 1 -- 172.21.15.90:0/600674287 >> v1:172.21.15.102:6789/0 conn(0x7f6f700cdf20 legacy=0x7f6f700ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f75200640 1 -- 172.21.15.90:0/600674287 >> v1:172.21.15.90:6789/0 conn(0x7f6f7006ce50 legacy=0x7f6f700ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.577+0000 7f6f75200640 1 -- 172.21.15.90:0/600674287 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f700cc710 con 0x7f6f7015e4e0 2024-03-31T15:54:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f75200640 1 --2- 172.21.15.90:0/600674287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 0x7f6f7006c910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097f0 tx=0x7f6f6401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52f233ec6a96638b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f6f400640 1 -- 172.21.15.90:0/600674287 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640b0070 con 0x7f6f7015e4e0 2024-03-31T15:54:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f6f400640 1 -- 172.21.15.90:0/600674287 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f640ab600 con 0x7f6f7015e4e0 2024-03-31T15:54:51.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f6f400640 1 -- 172.21.15.90:0/600674287 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640ab9d0 con 0x7f6f7015e4e0 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- 172.21.15.90:0/600674287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 msgr2=0x7f6f7006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 --2- 172.21.15.90:0/600674287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 0x7f6f7006c910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f640097f0 tx=0x7f6f6401c450 comp rx=0 tx=0).stop 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- 172.21.15.90:0/600674287 shutdown_connections 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 --2- 172.21.15.90:0/600674287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f7015e4e0 0x7f6f7006c910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- 172.21.15.90:0/600674287 >> 172.21.15.90:0/600674287 conn(0x7f6f700d1f20 msgr2=0x7f6f700d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- 172.21.15.90:0/600674287 shutdown_connections 2024-03-31T15:54:51.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- 172.21.15.90:0/600674287 wait complete. 2024-03-31T15:54:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 Processor -- start 2024-03-31T15:54:51.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.581+0000 7f6f77c00640 1 -- start start 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f700cdf20 0x7f6f701536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7015e4e0 0x7f6f700caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f700d0d90 con 0x7f6f7015e4e0 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f700d0c10 con 0x7f6f7006ce50 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7016c850 con 0x7f6f700cdf20 2024-03-31T15:54:51.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:51384/0 (socket says 172.21.15.90:51384) 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 -- 172.21.15.90:0/3450691915 learned_addr learned my addr 172.21.15.90:0/3450691915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7015e4e0 0x7f6f700caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f6fe00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f700cdf20 0x7f6f701536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 -- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f700cdf20 msgr2=0x7f6f701536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f700cdf20 0x7f6f701536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 -- 172.21.15.90:0/3450691915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7015e4e0 msgr2=0x7f6f700caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7015e4e0 0x7f6f700caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 -- 172.21.15.90:0/3450691915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f701540d0 con 0x7f6f7006ce50 2024-03-31T15:54:51.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f75200640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f64019060 tx=0x7f6f640191f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640bb070 con 0x7f6f7006ce50 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f640b7460 con 0x7f6f7006ce50 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f701543b0 con 0x7f6f7006ce50 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f640b0040 con 0x7f6f7006ce50 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7014c280 con 0x7f6f7006ce50 2024-03-31T15:54:51.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.585+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f700685f0 con 0x7f6f7006ce50 2024-03-31T15:54:51.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.589+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6f640be070 con 0x7f6f7006ce50 2024-03-31T15:54:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.589+0000 7f6f6cc00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 0x7f6f5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.589+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6f64121520 con 0x7f6f7006ce50 2024-03-31T15:54:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.589+0000 7f6f6fe00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 0x7f6f5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.589+0000 7f6f6fe00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 0x7f6f5006a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f7014fef0 tx=0x7f6f5c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.593+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6f640160d0 con 0x7f6f7006ce50 2024-03-31T15:54:51.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.749+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6f70156800 con 0x7f6f7006ce50 2024-03-31T15:54:51.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f6cc00640 1 -- 172.21.15.90:0/3450691915 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f6f640f1100 con 0x7f6f7006ce50 2024-03-31T15:54:51.744 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 msgr2=0x7f6f5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 0x7f6f5006a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f7014fef0 tx=0x7f6f5c007d40 comp rx=0 tx=0).stop 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 msgr2=0x7f6f70149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f64019060 tx=0x7f6f640191f0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 shutdown_connections 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f500683b0 0x7f6f5006a870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f7015e4e0 0x7f6f700caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f700cdf20 0x7f6f701536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 --2- 172.21.15.90:0/3450691915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f7006ce50 0x7f6f70149290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 >> 172.21.15.90:0/3450691915 conn(0x7f6f700d1f20 msgr2=0x7f6f700d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 shutdown_connections 2024-03-31T15:54:51.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.753+0000 7f6f77c00640 1 -- 172.21.15.90:0/3450691915 wait complete. 2024-03-31T15:54:51.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:51.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 Processor -- start 2024-03-31T15:54:51.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 -- start start 2024-03-31T15:54:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd99c15c290 0x7fd99c0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd99c0af210 0x7fd99c071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99c06e9c0 con 0x7fd99c0ad3d0 2024-03-31T15:54:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd99c06de40 con 0x7fd99c0af210 2024-03-31T15:54:51.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd9a1c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd99c06e120 con 0x7fd99c15c290 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99a000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd99c0ad3d0 0x7fd99c0aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44848/0 (socket says 172.21.15.90:44848) 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd99c0af210 0x7fd99c071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99a000640 1 -- 172.21.15.90:0/426953068 learned_addr learned my addr 172.21.15.90:0/426953068 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99aa00640 1 --2- 172.21.15.90:0/426953068 >> v2:172.21.15.102:3300/0 conn(0x7fd99c15c290 0x7fd99c0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99b400640 1 -- 172.21.15.90:0/426953068 >> v2:172.21.15.102:3300/0 conn(0x7fd99c15c290 msgr2=0x7fd99c0ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:51.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99b400640 1 --2- 172.21.15.90:0/426953068 >> v2:172.21.15.102:3300/0 conn(0x7fd99c15c290 0x7fd99c0ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99b400640 1 -- 172.21.15.90:0/426953068 >> v1:172.21.15.90:6789/0 conn(0x7fd99c0ad3d0 legacy=0x7fd99c0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd99b400640 1 -- 172.21.15.90:0/426953068 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c06e400 con 0x7fd99c0af210 2024-03-31T15:54:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.981+0000 7fd999600640 1 -- 172.21.15.90:0/426953068 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2852440576 0 0) 0x7fd99c06e9c0 con 0x7fd99c0ad3d0 2024-03-31T15:54:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd99b400640 1 --2- 172.21.15.90:0/426953068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c071750 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0cfa30 tx=0x7fd99401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcbf030b8e11ca server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd999600640 1 -- 172.21.15.90:0/426953068 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9940b2070 con 0x7fd99c0af210 2024-03-31T15:54:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd999600640 1 -- 172.21.15.90:0/426953068 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9940a57f0 con 0x7fd99c0af210 2024-03-31T15:54:51.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd999600640 1 -- 172.21.15.90:0/426953068 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9940a5bc0 con 0x7fd99c0af210 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 -- 172.21.15.90:0/426953068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 msgr2=0x7fd99c071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/426953068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c071750 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0cfa30 tx=0x7fd99401c930 comp rx=0 tx=0).stop 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 -- 172.21.15.90:0/426953068 shutdown_connections 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/426953068 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c071750 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/426953068 >> v2:172.21.15.102:3300/0 conn(0x7fd99c15c290 0x7fd99c0ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 -- 172.21.15.90:0/426953068 >> 172.21.15.90:0/426953068 conn(0x7fd99c0b18b0 msgr2=0x7fd99c0b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:51.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 -- 172.21.15.90:0/426953068 shutdown_connections 2024-03-31T15:54:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.985+0000 7fd9a1c00640 1 -- 172.21.15.90:0/426953068 wait complete. 2024-03-31T15:54:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 Processor -- start 2024-03-31T15:54:51.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 -- start start 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd99c0ad3d0 0x7fd99c152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd99c15c290 0x7fd99c14bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c072830 con 0x7fd99c0af210 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c0726b0 con 0x7fd99c0ad3d0 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd99c0729b0 con 0x7fd99c15c290 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37292/0 (socket says 172.21.15.90:37292) 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 -- 172.21.15.90:0/1996780604 learned_addr learned my addr 172.21.15.90:0/1996780604 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99aa00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd99c0ad3d0 0x7fd99c152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99b400640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd99c15c290 0x7fd99c14bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 -- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd99c15c290 msgr2=0x7fd99c14bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd99c15c290 0x7fd99c14bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 -- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd99c0ad3d0 msgr2=0x7fd99c152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd99c0ad3d0 0x7fd99c152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 -- 172.21.15.90:0/1996780604 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c15bb00 con 0x7fd99c0af210 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd99a000640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd98800c4b0 tx=0x7fd98800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988018070 con 0x7fd99c0af210 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd988007500 con 0x7fd99c0af210 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988004a30 con 0x7fd99c0af210 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.989+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c14b2c0 con 0x7fd99c0af210 2024-03-31T15:54:51.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c15bde0 con 0x7fd99c0af210 2024-03-31T15:54:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c075880 con 0x7fd99c0af210 2024-03-31T15:54:51.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd988004bd0 con 0x7fd99c0af210 2024-03-31T15:54:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd992a00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 0x7fd97c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd988086060 con 0x7fd99c0af210 2024-03-31T15:54:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd99aa00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 0x7fd97c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:51.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.993+0000 7fd99aa00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 0x7fd97c06a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009ac0 tx=0x7fd98c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:51.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:51.997+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd988055bf0 con 0x7fd99c0af210 2024-03-31T15:54:52.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd99c0685f0 con 0x7fd99c0af210 2024-03-31T15:54:52.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd992a00640 1 -- 172.21.15.90:0/1996780604 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd988059ef0 con 0x7fd99c0af210 2024-03-31T15:54:52.150 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 msgr2=0x7fd97c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 0x7fd97c06a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009ac0 tx=0x7fd98c009190 comp rx=0 tx=0).stop 2024-03-31T15:54:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 msgr2=0x7fd99c14ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd98800c4b0 tx=0x7fd98800c970 comp rx=0 tx=0).stop 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 shutdown_connections 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd97c068380 0x7fd97c06a840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd99c15c290 0x7fd99c14bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd99c0af210 0x7fd99c14ab10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 --2- 172.21.15.90:0/1996780604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd99c0ad3d0 0x7fd99c152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 >> 172.21.15.90:0/1996780604 conn(0x7fd99c0b18b0 msgr2=0x7fd99c0cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:52.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 shutdown_connections 2024-03-31T15:54:52.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.157+0000 7fd9a1c00640 1 -- 172.21.15.90:0/1996780604 wait complete. 2024-03-31T15:54:52.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:52.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.317+0000 7fc6abe00640 1 Processor -- start 2024-03-31T15:54:52.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.317+0000 7fc6abe00640 1 -- start start 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6abe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc6ac15b250 0x7fc6ac06e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6abe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ac0cd710 con 0x7fc6ac06ee00 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6abe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ac06f2a0 con 0x7fc6ac0b4230 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6abe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ac06f580 con 0x7fc6ac15b250 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6a2000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc6ac15b250 0x7fc6ac06e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6ab400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc6ac06ee00 0x7fc6ac0cded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44860/0 (socket says 172.21.15.90:44860) 2024-03-31T15:54:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6ab400640 1 -- 172.21.15.90:0/937644361 learned_addr learned my addr 172.21.15.90:0/937644361 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6a2000640 1 -- 172.21.15.90:0/937644361 >> v1:172.21.15.102:6789/0 conn(0x7fc6ac0b4230 legacy=0x7fc6ac0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6a2000640 1 -- 172.21.15.90:0/937644361 >> v1:172.21.15.90:6789/0 conn(0x7fc6ac06ee00 legacy=0x7fc6ac0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6aa000640 1 -- 172.21.15.90:0/937644361 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853601205 0 0) 0x7fc6ac0cd710 con 0x7fc6ac06ee00 2024-03-31T15:54:52.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.321+0000 7fc6a2000640 1 -- 172.21.15.90:0/937644361 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac06f860 con 0x7fc6ac15b250 2024-03-31T15:54:52.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6a2000640 1 --2- 172.21.15.90:0/937644361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac06e8c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc694009d90 tx=0x7fc69401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd172f30c5dd2a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6aa000640 1 -- 172.21.15.90:0/937644361 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940b2070 con 0x7fc6ac15b250 2024-03-31T15:54:52.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6aa000640 1 -- 172.21.15.90:0/937644361 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6940a59e0 con 0x7fc6ac15b250 2024-03-31T15:54:52.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6aa000640 1 -- 172.21.15.90:0/937644361 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940a5db0 con 0x7fc6ac15b250 2024-03-31T15:54:52.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- 172.21.15.90:0/937644361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 msgr2=0x7fc6ac06e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 --2- 172.21.15.90:0/937644361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac06e8c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc694009d90 tx=0x7fc69401c900 comp rx=0 tx=0).stop 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- 172.21.15.90:0/937644361 shutdown_connections 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 --2- 172.21.15.90:0/937644361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac06e8c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- 172.21.15.90:0/937644361 >> 172.21.15.90:0/937644361 conn(0x7fc6ac0d0040 msgr2=0x7fc6ac0d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- 172.21.15.90:0/937644361 shutdown_connections 2024-03-31T15:54:52.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- 172.21.15.90:0/937644361 wait complete. 2024-03-31T15:54:52.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 Processor -- start 2024-03-31T15:54:52.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- start start 2024-03-31T15:54:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 0x7fc6ac150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac0cf200 con 0x7fc6ac15b250 2024-03-31T15:54:52.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac0cf080 con 0x7fc6ac06ee00 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.325+0000 7fc6abe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ac0714b0 con 0x7fc6ac0b4230 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6ab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 0x7fc6ac150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6ab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37306/0 (socket says 172.21.15.90:37306) 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:60984/0 (socket says 172.21.15.90:60984) 2024-03-31T15:54:52.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6ab400640 1 -- 172.21.15.90:0/2022302170 learned_addr learned my addr 172.21.15.90:0/2022302170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 -- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 msgr2=0x7fc6ac166390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 -- 172.21.15.90:0/2022302170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 msgr2=0x7fc6ac1f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 -- 172.21.15.90:0/2022302170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac0c98a0 con 0x7fc6ac06ee00 2024-03-31T15:54:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a2000640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6ab400640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6aaa00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 0x7fc6ac150bd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c3e0 tx=0x7fc69c00c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c011790 con 0x7fc6ac06ee00 2024-03-31T15:54:52.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c017070 con 0x7fc6ac06ee00 2024-03-31T15:54:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac0c9b80 con 0x7fc6ac06ee00 2024-03-31T15:54:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c011bb0 con 0x7fc6ac06ee00 2024-03-31T15:54:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac0ca420 con 0x7fc6ac06ee00 2024-03-31T15:54:52.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac0ca620 con 0x7fc6ac06ee00 2024-03-31T15:54:52.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.329+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc69c01c460 con 0x7fc6ac06ee00 2024-03-31T15:54:52.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.333+0000 7fc6a3e00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 0x7fc68806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.333+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc69c086ac0 con 0x7fc6ac06ee00 2024-03-31T15:54:52.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.333+0000 7fc6a2000640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 0x7fc68806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.333+0000 7fc6a2000640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 0x7fc68806a840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc6940af000 tx=0x7fc6940a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.337+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc6ac0ca620 con 0x7fc6ac06ee00 2024-03-31T15:54:52.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.493+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6ac068d00 con 0x7fc6ac06ee00 2024-03-31T15:54:52.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.493+0000 7fc6a3e00640 1 -- 172.21.15.90:0/2022302170 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc69c056590 con 0x7fc6ac06ee00 2024-03-31T15:54:52.487 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:52.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 msgr2=0x7fc68806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 0x7fc68806a840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc6940af000 tx=0x7fc6940a7070 comp rx=0 tx=0).stop 2024-03-31T15:54:52.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 msgr2=0x7fc6ac150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 0x7fc6ac150bd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c3e0 tx=0x7fc69c00c8a0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 shutdown_connections 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc688068380 0x7fc68806a840 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6ac15b250 0x7fc6ac1f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6ac0b4230 0x7fc6ac166390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 --2- 172.21.15.90:0/2022302170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6ac06ee00 0x7fc6ac150bd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 >> 172.21.15.90:0/2022302170 conn(0x7fc6ac0d0040 msgr2=0x7fc6ac153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 shutdown_connections 2024-03-31T15:54:52.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.497+0000 7fc6abe00640 1 -- 172.21.15.90:0/2022302170 wait complete. 2024-03-31T15:54:52.508 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:52.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 Processor -- start 2024-03-31T15:54:52.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 -- start start 2024-03-31T15:54:52.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f655415e4e0 0x7f655406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655415e8c0 con 0x7f655406ce50 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65540d0ff0 con 0x7f65540cdf20 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f655ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f65540d12d0 con 0x7f655415e4e0 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6559000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f655415e4e0 0x7f655406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6553e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f655406ce50 0x7f65540ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44862/0 (socket says 172.21.15.90:44862) 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6553e00640 1 -- 172.21.15.90:0/2855418135 learned_addr learned my addr 172.21.15.90:0/2855418135 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6559000640 1 -- 172.21.15.90:0/2855418135 >> v1:172.21.15.102:6789/0 conn(0x7f65540cdf20 legacy=0x7f65540ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:52.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6559000640 1 -- 172.21.15.90:0/2855418135 >> v1:172.21.15.90:6789/0 conn(0x7f655406ce50 legacy=0x7f65540ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6559000640 1 -- 172.21.15.90:0/2855418135 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65540cc710 con 0x7f655415e4e0 2024-03-31T15:54:52.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.657+0000 7f6553400640 1 -- 172.21.15.90:0/2855418135 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511892657 0 0) 0x7f655415e8c0 con 0x7f655406ce50 2024-03-31T15:54:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f6559000640 1 --2- 172.21.15.90:0/2855418135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f655406c910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f65480097f0 tx=0x7f654801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea23ad006fdb05f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f6553400640 1 -- 172.21.15.90:0/2855418135 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480b0070 con 0x7f655415e4e0 2024-03-31T15:54:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f6553400640 1 -- 172.21.15.90:0/2855418135 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65480ab600 con 0x7f655415e4e0 2024-03-31T15:54:52.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f6553400640 1 -- 172.21.15.90:0/2855418135 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480a4e60 con 0x7f655415e4e0 2024-03-31T15:54:52.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- 172.21.15.90:0/2855418135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 msgr2=0x7f655406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 --2- 172.21.15.90:0/2855418135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f655406c910 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f65480097f0 tx=0x7f654801c3d0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- 172.21.15.90:0/2855418135 shutdown_connections 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 --2- 172.21.15.90:0/2855418135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f655406c910 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- 172.21.15.90:0/2855418135 >> 172.21.15.90:0/2855418135 conn(0x7f65540d1f20 msgr2=0x7f65540d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- 172.21.15.90:0/2855418135 shutdown_connections 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- 172.21.15.90:0/2855418135 wait complete. 2024-03-31T15:54:52.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 Processor -- start 2024-03-31T15:54:52.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.661+0000 7f655ba00640 1 -- start start 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f655406ce50 0x7f6554144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65540cdf20 0x7f65541554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65540d0d90 con 0x7f655415e4e0 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65540d0c10 con 0x7f655406ce50 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f655416c850 con 0x7f65540cdf20 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37310/0 (socket says 172.21.15.90:37310) 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 -- 172.21.15.90:0/1355921011 learned_addr learned my addr 172.21.15.90:0/1355921011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559000640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f655406ce50 0x7f6554144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6553e00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65540cdf20 0x7f65541554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 -- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65540cdf20 msgr2=0x7f65541554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65540cdf20 0x7f65541554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 -- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f655406ce50 msgr2=0x7f6554144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f655406ce50 0x7f6554144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 -- 172.21.15.90:0/1355921011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6554154410 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6559a00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c780 tx=0x7f654c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c004b70 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f654c004d10 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65541546f0 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c00aa90 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6554154cb0 con 0x7f655415e4e0 2024-03-31T15:54:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.665+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65540685f0 con 0x7f655415e4e0 2024-03-31T15:54:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.669+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f654c00ac30 con 0x7f655415e4e0 2024-03-31T15:54:52.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.669+0000 7f6550c00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 0x7f653406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:52.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.669+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f654c0871d0 con 0x7f655415e4e0 2024-03-31T15:54:52.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.669+0000 7f6559000640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 0x7f653406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:52.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.669+0000 7f6559000640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 0x7f653406a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f65480ad000 tx=0x7f65480a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:52.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.673+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f654c056d40 con 0x7f655415e4e0 2024-03-31T15:54:52.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f655415e8c0 con 0x7f655415e4e0 2024-03-31T15:54:52.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f6550c00640 1 -- 172.21.15.90:0/1355921011 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f654c05b040 con 0x7f655415e4e0 2024-03-31T15:54:52.826 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:52.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 msgr2=0x7f653406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 0x7f653406a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f65480ad000 tx=0x7f65480a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:52.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 msgr2=0x7f6554149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.833+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c780 tx=0x7f654c00cc40 comp rx=0 tx=0).stop 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 shutdown_connections 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6534068380 0x7f653406a840 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f655415e4e0 0x7f6554149250 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65540cdf20 0x7f65541554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 --2- 172.21.15.90:0/1355921011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f655406ce50 0x7f6554144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 >> 172.21.15.90:0/1355921011 conn(0x7f65540d1f20 msgr2=0x7f65540c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 shutdown_connections 2024-03-31T15:54:52.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:52.837+0000 7f655ba00640 1 -- 172.21.15.90:0/1355921011 wait complete. 2024-03-31T15:54:52.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:53.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 Processor -- start 2024-03-31T15:54:53.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 -- start start 2024-03-31T15:54:53.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7e80cdc10 0x7fe7e80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 0x7fe7e806f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e80d0210 con 0x7fe7e80ce710 2024-03-31T15:54:53.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e80d04f0 con 0x7fe7e80cdc10 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ede00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e80d07d0 con 0x7fe7e80d0fd0 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7eca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe7e80cdc10 0x7fe7e80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ed400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe7e80ce710 0x7fe7e815d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50138/0 (socket says 172.21.15.90:50138) 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7ed400640 1 -- 172.21.15.90:0/4066647382 learned_addr learned my addr 172.21.15.90:0/4066647382 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7dfe00640 1 --2- 172.21.15.90:0/4066647382 >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 0x7fe7e806f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7eca00640 1 -- 172.21.15.90:0/4066647382 >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 msgr2=0x7fe7e806f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7eca00640 1 --2- 172.21.15.90:0/4066647382 >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 0x7fe7e806f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7eca00640 1 -- 172.21.15.90:0/4066647382 >> v1:172.21.15.90:6789/0 conn(0x7fe7e80ce710 legacy=0x7fe7e815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7eca00640 1 -- 172.21.15.90:0/4066647382 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e80d0ab0 con 0x7fe7e80cdc10 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7dfe00640 1 --2- 172.21.15.90:0/4066647382 >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 0x7fe7e806f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:53.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.037+0000 7fe7e7e00640 1 -- 172.21.15.90:0/4066647382 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877790987 0 0) 0x7fe7e80d0210 con 0x7fe7e80ce710 2024-03-31T15:54:53.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7eca00640 1 --2- 172.21.15.90:0/4066647382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80b43e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8071f60 tx=0x7fe7d801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5031cd1d2a5c7a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7e7e00640 1 -- 172.21.15.90:0/4066647382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80b0070 con 0x7fe7e80cdc10 2024-03-31T15:54:53.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7e7e00640 1 -- 172.21.15.90:0/4066647382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d80ab600 con 0x7fe7e80cdc10 2024-03-31T15:54:53.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7e7e00640 1 -- 172.21.15.90:0/4066647382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80ab9d0 con 0x7fe7e80cdc10 2024-03-31T15:54:53.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 -- 172.21.15.90:0/4066647382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80cdc10 msgr2=0x7fe7e80b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 --2- 172.21.15.90:0/4066647382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80b43e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8071f60 tx=0x7fe7d801c3f0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 -- 172.21.15.90:0/4066647382 shutdown_connections 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 --2- 172.21.15.90:0/4066647382 >> v2:172.21.15.102:3300/0 conn(0x7fe7e80d0fd0 0x7fe7e806f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 --2- 172.21.15.90:0/4066647382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80b43e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 -- 172.21.15.90:0/4066647382 >> 172.21.15.90:0/4066647382 conn(0x7fe7e80b38c0 msgr2=0x7fe7e80b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 -- 172.21.15.90:0/4066647382 shutdown_connections 2024-03-31T15:54:53.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.041+0000 7fe7ede00640 1 -- 172.21.15.90:0/4066647382 wait complete. 2024-03-31T15:54:53.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 Processor -- start 2024-03-31T15:54:53.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- start start 2024-03-31T15:54:53.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 0x7fe7e80b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e80cc380 con 0x7fe7e80ce710 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e80ce200 con 0x7fe7e80cdc10 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e80cc500 con 0x7fe7e80d0fd0 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41506/0 (socket says 172.21.15.90:41506) 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 -- 172.21.15.90:0/2759206807 learned_addr learned my addr 172.21.15.90:0/2759206807 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7eca00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ed400640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 0x7fe7e80b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 -- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 msgr2=0x7fe7e80b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 0x7fe7e80b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 -- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e80cdc10 msgr2=0x7fe7e80caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 -- 172.21.15.90:0/2759206807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e815ec60 con 0x7fe7e80ce710 2024-03-31T15:54:53.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ed400640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 0x7fe7e80b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:53.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7dfe00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000d800 tx=0x7fe7d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0018020 con 0x7fe7e80ce710 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d00096c0 con 0x7fe7e80ce710 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e815ef40 con 0x7fe7e80ce710 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0009ab0 con 0x7fe7e80ce710 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.045+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e815f500 con 0x7fe7e80ce710 2024-03-31T15:54:53.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.049+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e80685f0 con 0x7fe7e80ce710 2024-03-31T15:54:53.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.049+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe7d001b020 con 0x7fe7e80ce710 2024-03-31T15:54:53.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.049+0000 7fe7e5600640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 0x7fe7c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.049+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe7d0087290 con 0x7fe7e80ce710 2024-03-31T15:54:53.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.049+0000 7fe7eca00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 0x7fe7c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.053+0000 7fe7eca00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 0x7fe7c406a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ad000 tx=0x7fe7d80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.053+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe7d0022020 con 0x7fe7e80ce710 2024-03-31T15:54:53.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.225+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe7e80b7bf0 con 0x7fe7e80ce710 2024-03-31T15:54:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.225+0000 7fe7e5600640 1 -- 172.21.15.90:0/2759206807 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fe7d0057450 con 0x7fe7e80ce710 2024-03-31T15:54:53.219 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:53.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 msgr2=0x7fe7c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 0x7fe7c406a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ad000 tx=0x7fe7d80a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 msgr2=0x7fe7e80afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7d000d800 tx=0x7fe7d000dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 shutdown_connections 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7c4068380 0x7fe7c406a840 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7e80d0fd0 0x7fe7e80b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7e80ce710 0x7fe7e80afd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 --2- 172.21.15.90:0/2759206807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e80cdc10 0x7fe7e80caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 >> 172.21.15.90:0/2759206807 conn(0x7fe7e80b38c0 msgr2=0x7fe7e806d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 shutdown_connections 2024-03-31T15:54:53.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.229+0000 7fe7ede00640 1 -- 172.21.15.90:0/2759206807 wait complete. 2024-03-31T15:54:53.248 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com... 2024-03-31T15:54:53.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:53.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-31T15:54:53.283 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:54:53.283 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:54:53.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:53.283 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:54:53.356 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:53.356 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:53.365 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:53.365 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:53.416 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 Processor -- start 2024-03-31T15:54:53.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 -- start start 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fac68145740 0x7fac68072ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac6815f9a0 0x7fac680cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 0x7fac680cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fac680cbc20 con 0x7fac6815f9a0 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fac681603f0 con 0x7fac68145740 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6dc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fac681606d0 con 0x7fac6815fe50 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac6815f9a0 0x7fac680cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac6815f9a0 0x7fac680cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41522/0 (socket says 172.21.15.90:41522) 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 -- 172.21.15.90:0/805354467 learned_addr learned my addr 172.21.15.90:0/805354467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac67e00640 1 --2- 172.21.15.90:0/805354467 >> v2:172.21.15.102:3300/0 conn(0x7fac68145740 0x7fac68072ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6d200640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 0x7fac680cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 -- 172.21.15.90:0/805354467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 msgr2=0x7fac680cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:53.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 0x7fac680cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 -- 172.21.15.90:0/805354467 >> v2:172.21.15.102:3300/0 conn(0x7fac68145740 msgr2=0x7fac68072ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:53.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 --2- 172.21.15.90:0/805354467 >> v2:172.21.15.102:3300/0 conn(0x7fac68145740 0x7fac68072ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac5fe00640 1 -- 172.21.15.90:0/805354467 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac680722d0 con 0x7fac6815f9a0 2024-03-31T15:54:53.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.605+0000 7fac6d200640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 0x7fac680cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:53.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac5fe00640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac6815f9a0 0x7fac680cb1a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac680ce310 tx=0x7fac5001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f4b38ba49ef1e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac67400640 1 -- 172.21.15.90:0/805354467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500b0070 con 0x7fac6815f9a0 2024-03-31T15:54:53.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac67400640 1 -- 172.21.15.90:0/805354467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac500ab600 con 0x7fac6815f9a0 2024-03-31T15:54:53.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac67400640 1 -- 172.21.15.90:0/805354467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac500ab9d0 con 0x7fac6815f9a0 2024-03-31T15:54:53.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- 172.21.15.90:0/805354467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac6815f9a0 msgr2=0x7fac680cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac6815f9a0 0x7fac680cb1a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fac680ce310 tx=0x7fac5001c520 comp rx=0 tx=0).stop 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- 172.21.15.90:0/805354467 shutdown_connections 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815fe50 0x7fac680cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 --2- 172.21.15.90:0/805354467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac6815f9a0 0x7fac680cb1a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 --2- 172.21.15.90:0/805354467 >> v2:172.21.15.102:3300/0 conn(0x7fac68145740 0x7fac68072ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- 172.21.15.90:0/805354467 >> 172.21.15.90:0/805354467 conn(0x7fac680cd660 msgr2=0x7fac680cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- 172.21.15.90:0/805354467 shutdown_connections 2024-03-31T15:54:53.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- 172.21.15.90:0/805354467 wait complete. 2024-03-31T15:54:53.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 Processor -- start 2024-03-31T15:54:53.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.609+0000 7fac6dc00640 1 -- start start 2024-03-31T15:54:53.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 0x7fac680ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 0x7fac68141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac68070b40 con 0x7fac68145740 2024-03-31T15:54:53.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac680709c0 con 0x7fac6815fe50 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fac6815c3a0 con 0x7fac6815f9a0 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41524/0 (socket says 172.21.15.90:41524) 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 -- 172.21.15.90:0/2067363257 learned_addr learned my addr 172.21.15.90:0/2067363257 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6d200640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 0x7fac68141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac5fe00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 0x7fac680ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 -- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 msgr2=0x7fac680ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 0x7fac680ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 -- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 msgr2=0x7fac68141df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 0x7fac68141df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 -- 172.21.15.90:0/2067363257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac681f8500 con 0x7fac68145740 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6d200640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 0x7fac68141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac5fe00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 0x7fac680ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:53.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac67e00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac5800d800 tx=0x7fac5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58018020 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac580096a0 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac681f87e0 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58009a50 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac681f8da0 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.613+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac68068d00 con 0x7fac68145740 2024-03-31T15:54:53.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.617+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fac5801b020 con 0x7fac68145740 2024-03-31T15:54:53.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.617+0000 7fac64c00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 0x7fac4406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.617+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fac580873b0 con 0x7fac68145740 2024-03-31T15:54:53.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.617+0000 7fac5fe00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 0x7fac4406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.617+0000 7fac5fe00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 0x7fac4406a840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac500ad000 tx=0x7fac50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.621+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fac58022020 con 0x7fac68145740 2024-03-31T15:54:53.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac680b6e70 con 0x7fac68145740 2024-03-31T15:54:53.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac64c00640 1 -- 172.21.15.90:0/2067363257 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7fac580574d0 con 0x7fac68145740 2024-03-31T15:54:53.774 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:53.774 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 msgr2=0x7fac4406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 0x7fac4406a840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fac500ad000 tx=0x7fac50008040 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 msgr2=0x7fac6813fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fac5800d800 tx=0x7fac5800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 shutdown_connections 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fac44068380 0x7fac4406a840 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6815fe50 0x7fac68141df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac6815f9a0 0x7fac680ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 --2- 172.21.15.90:0/2067363257 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac68145740 0x7fac6813fe80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 >> 172.21.15.90:0/2067363257 conn(0x7fac680cd660 msgr2=0x7fac680cb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 shutdown_connections 2024-03-31T15:54:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.781+0000 7fac6dc00640 1 -- 172.21.15.90:0/2067363257 wait complete. 2024-03-31T15:54:53.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:53.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:53.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 Processor -- start 2024-03-31T15:54:53.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 -- start start 2024-03-31T15:54:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 0x7f129c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f129c15e8c0 con 0x7f129c0ce7a0 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f129c0d0ff0 con 0x7f129c06ce50 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f12a1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f129c0d12d0 con 0x7f129c15e4e0 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f1299e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 0x7f129c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f1299e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:54242/0 (socket says 172.21.15.90:54242) 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f1299e00640 1 -- 172.21.15.90:0/2331967183 learned_addr learned my addr 172.21.15.90:0/2331967183 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129b200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f129c0ce7a0 0x7f129c0cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56852/0 (socket says 172.21.15.90:56852) 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129a800640 1 -- 172.21.15.90:0/2331967183 >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 msgr2=0x7f129c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:53.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129a800640 1 --2- 172.21.15.90:0/2331967183 >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129a800640 1 -- 172.21.15.90:0/2331967183 >> v1:172.21.15.102:6789/0 conn(0x7f129c0ce7a0 legacy=0x7f129c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f129a800640 1 -- 172.21.15.90:0/2331967183 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129c0cc710 con 0x7f129c15e4e0 2024-03-31T15:54:53.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.941+0000 7f1299e00640 1 --2- 172.21.15.90:0/2331967183 >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:53.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f129a800640 1 --2- 172.21.15.90:0/2331967183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 0x7f129c06c910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f128c0097c0 tx=0x7f128c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e98dda8119fba6f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f1299400640 1 -- 172.21.15.90:0/2331967183 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c0b0070 con 0x7f129c15e4e0 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f1299400640 1 -- 172.21.15.90:0/2331967183 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f128c0ab600 con 0x7f129c15e4e0 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f1299400640 1 -- 172.21.15.90:0/2331967183 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c0ab9d0 con 0x7f129c15e4e0 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 -- 172.21.15.90:0/2331967183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 msgr2=0x7f129c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 --2- 172.21.15.90:0/2331967183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 0x7f129c06c910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f128c0097c0 tx=0x7f128c01c410 comp rx=0 tx=0).stop 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 -- 172.21.15.90:0/2331967183 shutdown_connections 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 --2- 172.21.15.90:0/2331967183 >> v2:172.21.15.102:3300/0 conn(0x7f129c06ce50 0x7f129c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 --2- 172.21.15.90:0/2331967183 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c15e4e0 0x7f129c06c910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 -- 172.21.15.90:0/2331967183 >> 172.21.15.90:0/2331967183 conn(0x7f129c0d1f20 msgr2=0x7f129c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:53.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 -- 172.21.15.90:0/2331967183 shutdown_connections 2024-03-31T15:54:53.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.945+0000 7f12a1400640 1 -- 172.21.15.90:0/2331967183 wait complete. 2024-03-31T15:54:53.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 Processor -- start 2024-03-31T15:54:53.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- start start 2024-03-31T15:54:53.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c06ce50 0x7f129c14d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129c15e4e0 0x7f129c0ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f129c0d0d90 con 0x7f129c0ce7a0 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f129c0d0c10 con 0x7f129c15e4e0 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f129c16b840 con 0x7f129c06ce50 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41540/0 (socket says 172.21.15.90:41540) 2024-03-31T15:54:53.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 -- 172.21.15.90:0/697672737 learned_addr learned my addr 172.21.15.90:0/697672737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f129b200640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129c15e4e0 0x7f129c0ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f129a800640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c06ce50 0x7f129c14d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 -- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c06ce50 msgr2=0x7f129c14d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c06ce50 0x7f129c14d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 -- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129c15e4e0 msgr2=0x7f129c0ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:53.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129c15e4e0 0x7f129c0ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 -- 172.21.15.90:0/697672737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129c0cc000 con 0x7f129c0ce7a0 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1299e00640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f128800e930 tx=0x7f128800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288017070 con 0x7f129c0ce7a0 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f128800ca00 con 0x7f129c0ce7a0 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129c0caa20 con 0x7f129c0ce7a0 2024-03-31T15:54:53.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128800cdd0 con 0x7f129c0ce7a0 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.949+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f129c0cafe0 con 0x7f129c0ce7a0 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f129c0685f0 con 0x7f129c0ce7a0 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f128801a070 con 0x7f129c0ce7a0 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f1292a00640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 0x7f127c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1288087520 con 0x7f129c0ce7a0 2024-03-31T15:54:53.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f129a800640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 0x7f127c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:53.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.953+0000 7f129a800640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 0x7f127c06a870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f128c0ad000 tx=0x7f128c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:53.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:53.957+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f129c0cc000 con 0x7f129c0ce7a0 2024-03-31T15:54:54.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.117+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f129c0c6910 con 0x7f129c0ce7a0 2024-03-31T15:54:54.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.117+0000 7f1292a00640 1 -- 172.21.15.90:0/697672737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f1288057640 con 0x7f129c0ce7a0 2024-03-31T15:54:54.111 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:54.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 msgr2=0x7f127c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 0x7f127c06a870 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f128c0ad000 tx=0x7f128c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 msgr2=0x7f129c0cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f128800e930 tx=0x7f128800edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 shutdown_connections 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f127c0683b0 0x7f127c06a870 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f129c15e4e0 0x7f129c0ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f129c0ce7a0 0x7f129c0cbac0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 --2- 172.21.15.90:0/697672737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f129c06ce50 0x7f129c14d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 >> 172.21.15.90:0/697672737 conn(0x7f129c0d1f20 msgr2=0x7f129c0c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 shutdown_connections 2024-03-31T15:54:54.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.121+0000 7f12a1400640 1 -- 172.21.15.90:0/697672737 wait complete. 2024-03-31T15:54:54.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:54.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 Processor -- start 2024-03-31T15:54:54.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 -- start start 2024-03-31T15:54:54.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8440cd640 0x7fa8440af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8440cefc0 con 0x7fa8440d2b20 2024-03-31T15:54:54.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa8440aed70 con 0x7fa8440cd640 2024-03-31T15:54:54.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.281+0000 7fa84b000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa8440cf2a0 con 0x7fa84415c320 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8440cd640 0x7fa8440af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa843e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa83be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa8440d2b20 0x7fa8440ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56858/0 (socket says 172.21.15.90:56858) 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa83be00640 1 -- 172.21.15.90:0/1674054591 learned_addr learned my addr 172.21.15.90:0/1674054591 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 -- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 msgr2=0x7fa8440cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 --2- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 -- 172.21.15.90:0/1674054591 >> v1:172.21.15.102:6789/0 conn(0x7fa8440d2b20 legacy=0x7fa8440ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 -- 172.21.15.90:0/1674054591 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8440cf580 con 0x7fa8440cd640 2024-03-31T15:54:54.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa843400640 1 -- 172.21.15.90:0/1674054591 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524262005 0 0) 0x7fa8440cefc0 con 0x7fa8440d2b20 2024-03-31T15:54:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa849000640 1 --2- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440cd640 0x7fa8440af200 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009d90 tx=0x7fa83c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ff2f7939126622 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.285+0000 7fa843400640 1 -- 172.21.15.90:0/1674054591 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0b2070 con 0x7fa8440cd640 2024-03-31T15:54:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843400640 1 -- 172.21.15.90:0/1674054591 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8440d0f20 con 0x7fa8440cd640 2024-03-31T15:54:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843400640 1 -- 172.21.15.90:0/1674054591 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0a5ad0 con 0x7fa8440cd640 2024-03-31T15:54:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440cd640 msgr2=0x7fa8440af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440cd640 0x7fa8440af200 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009d90 tx=0x7fa83c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- 172.21.15.90:0/1674054591 shutdown_connections 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440cd640 0x7fa8440af200 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- 172.21.15.90:0/1674054591 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- 172.21.15.90:0/1674054591 >> 172.21.15.90:0/1674054591 conn(0x7fa8440d1f20 msgr2=0x7fa8440d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- 172.21.15.90:0/1674054591 shutdown_connections 2024-03-31T15:54:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- 172.21.15.90:0/1674054591 wait complete. 2024-03-31T15:54:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 Processor -- start 2024-03-31T15:54:54.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- start start 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440d2b20 0x7fa8441550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa8440b0360 con 0x7fa8440cd640 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa8440b01e0 con 0x7fa8440d2b20 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa84b000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa8440b04e0 con 0x7fa84415c320 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41544/0 (socket says 172.21.15.90:41544) 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 -- 172.21.15.90:0/4283925337 learned_addr learned my addr 172.21.15.90:0/4283925337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa83be00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440d2b20 0x7fa8441550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa849000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 -- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 msgr2=0x7fa8440cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 -- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440d2b20 msgr2=0x7fa8441550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.289+0000 7fa843e00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440d2b20 0x7fa8441550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa843e00640 1 -- 172.21.15.90:0/4283925337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa844154ed0 con 0x7fa8440cd640 2024-03-31T15:54:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa843e00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa83400e930 tx=0x7fa83400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834017070 con 0x7fa8440cd640 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83400c9a0 con 0x7fa8440cd640 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83400cd50 con 0x7fa8440cd640 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8440c93b0 con 0x7fa8440cd640 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8440c9970 con 0x7fa8440cd640 2024-03-31T15:54:54.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa84415d3d0 con 0x7fa8440cd640 2024-03-31T15:54:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa83401a070 con 0x7fa8440cd640 2024-03-31T15:54:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 0x7fa82006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa834086d80 con 0x7fa8440cd640 2024-03-31T15:54:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.293+0000 7fa83be00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 0x7fa82006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.297+0000 7fa83be00640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 0x7fa82006a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa8440c55e0 tx=0x7fa82c007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.297+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa834057000 con 0x7fa8440cd640 2024-03-31T15:54:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa8440685f0 con 0x7fa8440cd640 2024-03-31T15:54:54.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa840c00640 1 -- 172.21.15.90:0/4283925337 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa834056900 con 0x7fa8440cd640 2024-03-31T15:54:54.449 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 msgr2=0x7fa82006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 0x7fa82006a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa8440c55e0 tx=0x7fa82c007d10 comp rx=0 tx=0).stop 2024-03-31T15:54:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 msgr2=0x7fa844154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa83400e930 tx=0x7fa83400edf0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 shutdown_connections 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa820068380 0x7fa82006a840 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa84415c320 0x7fa8440cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8440d2b20 0x7fa8441550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 --2- 172.21.15.90:0/4283925337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa8440cd640 0x7fa844154990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 >> 172.21.15.90:0/4283925337 conn(0x7fa8440d1f20 msgr2=0x7fa844070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 shutdown_connections 2024-03-31T15:54:54.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.457+0000 7fa84b000640 1 -- 172.21.15.90:0/4283925337 wait complete. 2024-03-31T15:54:54.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.613+0000 7fb061a00640 1 Processor -- start 2024-03-31T15:54:54.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.613+0000 7fb061a00640 1 -- start start 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 0x7fb05c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb05c06ce50 0x7fb05c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c15e8c0 con 0x7fb05c06ce50 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb05c0d0ff0 con 0x7fb05c15e4e0 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb061a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c0d12d0 con 0x7fb05c0ce7a0 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb05c06ce50 0x7fb05c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb05c06ce50 0x7fb05c06d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41548/0 (socket says 172.21.15.90:41548) 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 -- 172.21.15.90:0/3675735366 learned_addr learned my addr 172.21.15.90:0/3675735366 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:54.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05aa00640 1 --2- 172.21.15.90:0/3675735366 >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 0x7fb05c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05b400640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 -- 172.21.15.90:0/3675735366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 msgr2=0x7fb05c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c0cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 -- 172.21.15.90:0/3675735366 >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 msgr2=0x7fb05c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 --2- 172.21.15.90:0/3675735366 >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 0x7fb05c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05a000640 1 -- 172.21.15.90:0/3675735366 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c0cc710 con 0x7fb05c06ce50 2024-03-31T15:54:54.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.617+0000 7fb05aa00640 1 --2- 172.21.15.90:0/3675735366 >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 0x7fb05c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:54.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb05a000640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c06d2c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb048009870 tx=0x7fb04801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53115c2c6530c62b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb059600640 1 -- 172.21.15.90:0/3675735366 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0480b2070 con 0x7fb05c06ce50 2024-03-31T15:54:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb059600640 1 -- 172.21.15.90:0/3675735366 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0480a9070 con 0x7fb05c06ce50 2024-03-31T15:54:54.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb059600640 1 -- 172.21.15.90:0/3675735366 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0480ad880 con 0x7fb05c06ce50 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 -- 172.21.15.90:0/3675735366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 msgr2=0x7fb05c06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c06d2c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb048009870 tx=0x7fb04801c490 comp rx=0 tx=0).stop 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 -- 172.21.15.90:0/3675735366 shutdown_connections 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c0cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 --2- 172.21.15.90:0/3675735366 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c06d2c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 --2- 172.21.15.90:0/3675735366 >> v2:172.21.15.102:3300/0 conn(0x7fb05c15e4e0 0x7fb05c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 -- 172.21.15.90:0/3675735366 >> 172.21.15.90:0/3675735366 conn(0x7fb05c0d1f20 msgr2=0x7fb05c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 -- 172.21.15.90:0/3675735366 shutdown_connections 2024-03-31T15:54:54.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.621+0000 7fb061a00640 1 -- 172.21.15.90:0/3675735366 wait complete. 2024-03-31T15:54:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 Processor -- start 2024-03-31T15:54:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- start start 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 0x7fb05c15f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c0cd090 con 0x7fb05c06ce50 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c0ccf10 con 0x7fb05c15e4e0 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb05c0cd210 con 0x7fb05c0ce7a0 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41560/0 (socket says 172.21.15.90:41560) 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 -- 172.21.15.90:0/2139916890 learned_addr learned my addr 172.21.15.90:0/2139916890 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05b400640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 0x7fb05c15f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05a000640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 -- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 msgr2=0x7fb05c157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 -- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 msgr2=0x7fb05c15f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 0x7fb05c15f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 -- 172.21.15.90:0/2139916890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c160440 con 0x7fb05c06ce50 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05b400640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 0x7fb05c15f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05a000640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb05aa00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00d800 tx=0x7fb04c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00b6c0 con 0x7fb05c06ce50 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c019070 con 0x7fb05c06ce50 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c00baa0 con 0x7fb05c06ce50 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c160720 con 0x7fb05c06ce50 2024-03-31T15:54:54.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.625+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c160ce0 con 0x7fb05c06ce50 2024-03-31T15:54:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.629+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c0685f0 con 0x7fb05c06ce50 2024-03-31T15:54:54.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.629+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb04c00bc40 con 0x7fb05c06ce50 2024-03-31T15:54:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.629+0000 7fb052a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 0x7fb03c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.629+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb04c0871f0 con 0x7fb05c06ce50 2024-03-31T15:54:54.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.629+0000 7fb05a000640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 0x7fb03c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.633+0000 7fb05a000640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 0x7fb03c06a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb0480af000 tx=0x7fb048008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.633+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb04c057320 con 0x7fb05c06ce50 2024-03-31T15:54:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb05c0c8050 con 0x7fb05c06ce50 2024-03-31T15:54:54.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb052a00640 1 -- 172.21.15.90:0/2139916890 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb04c056c20 con 0x7fb05c06ce50 2024-03-31T15:54:54.786 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 msgr2=0x7fb03c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 0x7fb03c06a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb0480af000 tx=0x7fb048008040 comp rx=0 tx=0).stop 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 msgr2=0x7fb05c1614e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00d800 tx=0x7fb04c00dcc0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 shutdown_connections 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb03c068380 0x7fb03c06a840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05c15e4e0 0x7fb05c15f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb05c0ce7a0 0x7fb05c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 --2- 172.21.15.90:0/2139916890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb05c06ce50 0x7fb05c1614e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.793+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 >> 172.21.15.90:0/2139916890 conn(0x7fb05c0d1f20 msgr2=0x7fb05c15d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.797+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 shutdown_connections 2024-03-31T15:54:54.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.797+0000 7fb061a00640 1 -- 172.21.15.90:0/2139916890 wait complete. 2024-03-31T15:54:54.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:54.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.953+0000 7f93f6000640 1 Processor -- start 2024-03-31T15:54:54.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.953+0000 7f93f6000640 1 -- start start 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93f6000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 0x7f93f006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93f6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93f00cdf20 0x7f93f006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93f6000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f015e8c0 con 0x7f93f006ce50 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93f6000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f93f00d0ff0 con 0x7f93f00cdf20 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93f6000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f93f00d12d0 con 0x7f93f015e4e0 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93f00cdf20 0x7f93f006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93f00cdf20 0x7f93f006d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41576/0 (socket says 172.21.15.90:41576) 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 -- 172.21.15.90:0/3444905608 learned_addr learned my addr 172.21.15.90:0/3444905608 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:54.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93eea00640 1 --1- 172.21.15.90:0/3444905608 >> v1:172.21.15.90:6789/0 conn(0x7f93f006ce50 0x7f93f00ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50152/0 (socket says 172.21.15.90:50152) 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93ef400640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 0x7f93f006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 -- 172.21.15.90:0/3444905608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 msgr2=0x7f93f006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 0x7f93f006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 -- 172.21.15.90:0/3444905608 >> v1:172.21.15.90:6789/0 conn(0x7f93f006ce50 legacy=0x7f93f00ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93efe00640 1 -- 172.21.15.90:0/3444905608 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f00cc710 con 0x7f93f00cdf20 2024-03-31T15:54:54.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.957+0000 7f93ef400640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 0x7f93f006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93efe00640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f00cdf20 0x7f93f006d290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009e50 tx=0x7f93e801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd29af7b22dc0d7d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ee000640 1 -- 172.21.15.90:0/3444905608 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80a5660 con 0x7f93f00cdf20 2024-03-31T15:54:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ee000640 1 -- 172.21.15.90:0/3444905608 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e80a5800 con 0x7f93f00cdf20 2024-03-31T15:54:54.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ee000640 1 -- 172.21.15.90:0/3444905608 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80a5bd0 con 0x7f93f00cdf20 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- 172.21.15.90:0/3444905608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f00cdf20 msgr2=0x7f93f006d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f00cdf20 0x7f93f006d290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009e50 tx=0x7f93e801c9a0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- 172.21.15.90:0/3444905608 shutdown_connections 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f00cdf20 0x7f93f006d290 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- 172.21.15.90:0/3444905608 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f015e4e0 0x7f93f006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- 172.21.15.90:0/3444905608 >> 172.21.15.90:0/3444905608 conn(0x7f93f00d1f20 msgr2=0x7f93f00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:54.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- 172.21.15.90:0/3444905608 shutdown_connections 2024-03-31T15:54:54.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- 172.21.15.90:0/3444905608 wait complete. 2024-03-31T15:54:54.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 Processor -- start 2024-03-31T15:54:54.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- start start 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 0x7f93f0150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 0x7f93f0155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f93f016b1b0 con 0x7f93f006ce50 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f93f016b030 con 0x7f93f015e4e0 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93f6000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f93f016b330 con 0x7f93f00cdf20 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ef400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41588/0 (socket says 172.21.15.90:41588) 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.961+0000 7f93ef400640 1 -- 172.21.15.90:0/2857624070 learned_addr learned my addr 172.21.15.90:0/2857624070 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93eea00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 0x7f93f0150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93efe00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 0x7f93f0155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 -- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 msgr2=0x7f93f0150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 0x7f93f0150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 -- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 msgr2=0x7f93f0155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 0x7f93f0155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 -- 172.21.15.90:0/2857624070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f014fb40 con 0x7f93f006ce50 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93efe00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 0x7f93f0155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93eea00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 0x7f93f0150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93ef400640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c820 tx=0x7f93e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0015070 con 0x7f93f006ce50 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e0007a30 con 0x7f93f006ce50 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f014fdc0 con 0x7f93f006ce50 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0007e40 con 0x7f93f006ce50 2024-03-31T15:54:54.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f0150360 con 0x7f93f006ce50 2024-03-31T15:54:54.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f00685f0 con 0x7f93f006ce50 2024-03-31T15:54:54.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f93e0018070 con 0x7f93f006ce50 2024-03-31T15:54:54.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.965+0000 7f93e7400640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 0x7f93d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:54.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.969+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f93e0086bd0 con 0x7f93f006ce50 2024-03-31T15:54:54.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.969+0000 7f93eea00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 0x7f93d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:54.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.969+0000 7f93eea00640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 0x7f93d006a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f93dc007fd0 tx=0x7f93dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:54.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:54.973+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f93f014fb40 con 0x7f93f006ce50 2024-03-31T15:54:55.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.149+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f93f00c6dd0 con 0x7f93f006ce50 2024-03-31T15:54:55.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.149+0000 7f93e7400640 1 -- 172.21.15.90:0/2857624070 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f93e0056e20 con 0x7f93f006ce50 2024-03-31T15:54:55.143 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 msgr2=0x7f93d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 0x7f93d006a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f93dc007fd0 tx=0x7f93dc009040 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 msgr2=0x7f93f014ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c820 tx=0x7f93e000cce0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 shutdown_connections 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93d0068380 0x7f93d006a840 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93f015e4e0 0x7f93f0155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93f00cdf20 0x7f93f0150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 --2- 172.21.15.90:0/2857624070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93f006ce50 0x7f93f014ccc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 >> 172.21.15.90:0/2857624070 conn(0x7f93f00d1f20 msgr2=0x7f93f00ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:55.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 shutdown_connections 2024-03-31T15:54:55.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.153+0000 7f93f6000640 1 -- 172.21.15.90:0/2857624070 wait complete. 2024-03-31T15:54:55.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:55.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.341+0000 7fd5e4e00640 1 Processor -- start 2024-03-31T15:54:55.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.341+0000 7fd5e4e00640 1 -- start start 2024-03-31T15:54:55.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5e4e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5e015be10 0x7fd5e00cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5e4e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e00cc840 con 0x7fd5e015c1f0 2024-03-31T15:54:55.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5e4e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e00ccae0 con 0x7fd5e00b43e0 2024-03-31T15:54:55.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5e4e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e00cf740 con 0x7fd5e015be10 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5df400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5e015be10 0x7fd5e00cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5d7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5e015c1f0 0x7fd5e00ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50164/0 (socket says 172.21.15.90:50164) 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5d7e00640 1 -- 172.21.15.90:0/3167092382 learned_addr learned my addr 172.21.15.90:0/3167092382 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5df400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5e015be10 0x7fd5e00cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41602/0 (socket says 172.21.15.90:41602) 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5df400640 1 -- 172.21.15.90:0/3167092382 >> v1:172.21.15.102:6789/0 conn(0x7fd5e00b43e0 legacy=0x7fd5e00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5df400640 1 -- 172.21.15.90:0/3167092382 >> v1:172.21.15.90:6789/0 conn(0x7fd5e015c1f0 legacy=0x7fd5e00ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5dea00640 1 -- 172.21.15.90:0/3167092382 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941311100 0 0) 0x7fd5e00cc840 con 0x7fd5e015c1f0 2024-03-31T15:54:55.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.345+0000 7fd5df400640 1 -- 172.21.15.90:0/3167092382 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e00ccdc0 con 0x7fd5e015be10 2024-03-31T15:54:55.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5df400640 1 --2- 172.21.15.90:0/3167092382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e015be10 0x7fd5e00cd8a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00097c0 tx=0x7fd5d001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4cd2efbe2897b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5dea00640 1 -- 172.21.15.90:0/3167092382 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00b1070 con 0x7fd5e015be10 2024-03-31T15:54:55.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5dea00640 1 -- 172.21.15.90:0/3167092382 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d00a8050 con 0x7fd5e015be10 2024-03-31T15:54:55.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5dea00640 1 -- 172.21.15.90:0/3167092382 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00ac860 con 0x7fd5e015be10 2024-03-31T15:54:55.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3167092382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e015be10 msgr2=0x7fd5e00cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3167092382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e015be10 0x7fd5e00cd8a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00097c0 tx=0x7fd5d001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3167092382 shutdown_connections 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3167092382 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e015be10 0x7fd5e00cd8a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3167092382 >> 172.21.15.90:0/3167092382 conn(0x7fd5e00b38c0 msgr2=0x7fd5e00b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3167092382 shutdown_connections 2024-03-31T15:54:55.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3167092382 wait complete. 2024-03-31T15:54:55.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 Processor -- start 2024-03-31T15:54:55.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.349+0000 7fd5e4e00640 1 -- start start 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 0x7fd5e006c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 0x7fd5e01541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e006fb30 con 0x7fd5e00b43e0 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e006fcb0 con 0x7fd5e006cc40 2024-03-31T15:54:55.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e006fe30 con 0x7fd5e015c1f0 2024-03-31T15:54:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41614/0 (socket says 172.21.15.90:41614) 2024-03-31T15:54:55.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 -- 172.21.15.90:0/3650988424 learned_addr learned my addr 172.21.15.90:0/3650988424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 -- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 msgr2=0x7fd5e006c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 0x7fd5e006c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5dfe00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 0x7fd5e01541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 0x7fd5e006c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 -- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 msgr2=0x7fd5e01541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 0x7fd5e01541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 -- 172.21.15.90:0/3650988424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e006d150 con 0x7fd5e00b43e0 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 0x7fd5e006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:55.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5dfe00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 0x7fd5e01541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5df400640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00af040 tx=0x7fd5d001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00b1050 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d00bd070 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00b88b0 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e006d430 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e015ca70 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e00685f0 con 0x7fd5e00b43e0 2024-03-31T15:54:55.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.353+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd5d00b8a50 con 0x7fd5e00b43e0 2024-03-31T15:54:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.357+0000 7fd5d7400640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 0x7fd5bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.357+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd5d0121730 con 0x7fd5e00b43e0 2024-03-31T15:54:55.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.357+0000 7fd5d7e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 0x7fd5bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.357+0000 7fd5d7e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 0x7fd5bc06a840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0154d70 tx=0x7fd5c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.361+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd5d0016090 con 0x7fd5e00b43e0 2024-03-31T15:54:55.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.533+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd5e00d1b10 con 0x7fd5e00b43e0 2024-03-31T15:54:55.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5d7400640 1 -- 172.21.15.90:0/3650988424 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fd5d00f18e0 con 0x7fd5e00b43e0 2024-03-31T15:54:55.528 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:55.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 msgr2=0x7fd5bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 0x7fd5bc06a840 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0154d70 tx=0x7fd5c8009040 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 msgr2=0x7fd5e0153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00af040 tx=0x7fd5d001ac20 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 shutdown_connections 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd5bc068380 0x7fd5bc06a840 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5e006cc40 0x7fd5e01541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5e015c1f0 0x7fd5e006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 --2- 172.21.15.90:0/3650988424 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5e00b43e0 0x7fd5e0153580 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 >> 172.21.15.90:0/3650988424 conn(0x7fd5e00b38c0 msgr2=0x7fd5e00aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 shutdown_connections 2024-03-31T15:54:55.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.537+0000 7fd5e4e00640 1 -- 172.21.15.90:0/3650988424 wait complete. 2024-03-31T15:54:55.559 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:54:55.559 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:55.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.801+0000 7fb07b800640 1 Processor -- start 2024-03-31T15:54:55.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.801+0000 7fb07b800640 1 -- start start 2024-03-31T15:54:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb07b800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0740cd140 0x7fb0740d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb07b800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb074160350 con 0x7fb07415fd00 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb07b800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb074160630 con 0x7fb07415c120 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb07b800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb074160bf0 con 0x7fb0740cd140 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb073e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb07415fd00 0x7fb0740cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50176/0 (socket says 172.21.15.90:50176) 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb073e00640 1 -- 172.21.15.90:0/832366307 learned_addr learned my addr 172.21.15.90:0/832366307 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb079800640 1 --2- 172.21.15.90:0/832366307 >> v2:172.21.15.102:3300/0 conn(0x7fb0740cd140 0x7fb0740d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb073400640 1 -- 172.21.15.90:0/832366307 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1574837175 0 0) 0x7fb074160350 con 0x7fb07415fd00 2024-03-31T15:54:55.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb073400640 1 -- 172.21.15.90:0/832366307 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb05c003530 con 0x7fb07415fd00 2024-03-31T15:54:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb079800640 1 -- 172.21.15.90:0/832366307 >> v1:172.21.15.102:6789/0 conn(0x7fb07415c120 legacy=0x7fb0740b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb079800640 1 -- 172.21.15.90:0/832366307 >> v1:172.21.15.90:6789/0 conn(0x7fb07415fd00 legacy=0x7fb0740cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.805+0000 7fb079800640 1 -- 172.21.15.90:0/832366307 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb074160910 con 0x7fb0740cd140 2024-03-31T15:54:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb079800640 1 --2- 172.21.15.90:0/832366307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0740d2680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009d90 tx=0x7fb06c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45f363a3c458c61c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb073400640 1 -- 172.21.15.90:0/832366307 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0b2070 con 0x7fb0740cd140 2024-03-31T15:54:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb073400640 1 -- 172.21.15.90:0/832366307 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06c0a58b0 con 0x7fb0740cd140 2024-03-31T15:54:55.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb073400640 1 -- 172.21.15.90:0/832366307 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0a5c80 con 0x7fb0740cd140 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- 172.21.15.90:0/832366307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 msgr2=0x7fb0740d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 --2- 172.21.15.90:0/832366307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0740d2680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009d90 tx=0x7fb06c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- 172.21.15.90:0/832366307 shutdown_connections 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 --2- 172.21.15.90:0/832366307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0740d2680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- 172.21.15.90:0/832366307 >> 172.21.15.90:0/832366307 conn(0x7fb07409aa40 msgr2=0x7fb07409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- 172.21.15.90:0/832366307 shutdown_connections 2024-03-31T15:54:55.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- 172.21.15.90:0/832366307 wait complete. 2024-03-31T15:54:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 Processor -- start 2024-03-31T15:54:55.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.809+0000 7fb07b800640 1 -- start start 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0741535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb07415c120 0x7fb0740c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb07416a1d0 con 0x7fb07415fd00 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb07416a050 con 0x7fb0740cd140 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb07416a350 con 0x7fb07415c120 2024-03-31T15:54:55.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41628/0 (socket says 172.21.15.90:41628) 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 -- 172.21.15.90:0/1684866368 learned_addr learned my addr 172.21.15.90:0/1684866368 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb078e00640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0741535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb073e00640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb07415c120 0x7fb0740c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 -- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb07415c120 msgr2=0x7fb0740c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb07415c120 0x7fb0740c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 -- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 msgr2=0x7fb0741535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0741535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 -- 172.21.15.90:0/1684866368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0740c6a00 con 0x7fb07415fd00 2024-03-31T15:54:55.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb079800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0b0040 tx=0x7fb06c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0b2050 con 0x7fb07415fd00 2024-03-31T15:54:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06c0be070 con 0x7fb07415fd00 2024-03-31T15:54:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0740c6ce0 con 0x7fb07415fd00 2024-03-31T15:54:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c0ad850 con 0x7fb07415fd00 2024-03-31T15:54:55.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0740c72a0 con 0x7fb07415fd00 2024-03-31T15:54:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0740686d0 con 0x7fb07415fd00 2024-03-31T15:54:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.813+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb06c0ad9f0 con 0x7fb07415fd00 2024-03-31T15:54:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.817+0000 7fb070c00640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 0x7fb05406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:55.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.817+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb06c121d70 con 0x7fb07415fd00 2024-03-31T15:54:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.817+0000 7fb078e00640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 0x7fb05406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:55.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.817+0000 7fb078e00640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 0x7fb05406a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb068007fd0 tx=0x7fb068009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:55.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.821+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb06c016560 con 0x7fb07415fd00 2024-03-31T15:54:55.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.977+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0740b7a00 con 0x7fb07415fd00 2024-03-31T15:54:55.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.977+0000 7fb070c00640 1 -- 172.21.15.90:0/1684866368 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7fb06c0f17c0 con 0x7fb07415fd00 2024-03-31T15:54:55.972 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:55.972 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 msgr2=0x7fb05406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 0x7fb05406a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb068007fd0 tx=0x7fb068009040 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 msgr2=0x7fb07406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0b0040 tx=0x7fb06c019680 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 shutdown_connections 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb054068380 0x7fb05406a840 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb07415fd00 0x7fb07406fa60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb07415c120 0x7fb0740c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 --2- 172.21.15.90:0/1684866368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0740cd140 0x7fb0741535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 >> 172.21.15.90:0/1684866368 conn(0x7fb07409aa40 msgr2=0x7fb074156fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:55.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 shutdown_connections 2024-03-31T15:54:55.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:55.981+0000 7fb07b800640 1 -- 172.21.15.90:0/1684866368 wait complete. 2024-03-31T15:54:56.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:56.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:56.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 Processor -- start 2024-03-31T15:54:56.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 -- start start 2024-03-31T15:54:56.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe0e015bbb0 0x7fe0e015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0e015bf90 0x7fe0e0071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e015f720 con 0x7fe0e00b4c20 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e00b2d30 con 0x7fe0e015bf90 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0e5c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e00b3010 con 0x7fe0e015bbb0 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0e015bf90 0x7fe0e0071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe0e015bf90 0x7fe0e0071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41640/0 (socket says 172.21.15.90:41640) 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 -- 172.21.15.90:0/1996802818 learned_addr learned my addr 172.21.15.90:0/1996802818 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0dea00640 1 --2- 172.21.15.90:0/1996802818 >> v2:172.21.15.102:3300/0 conn(0x7fe0e015bbb0 0x7fe0e015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 -- 172.21.15.90:0/1996802818 >> v2:172.21.15.102:3300/0 conn(0x7fe0e015bbb0 msgr2=0x7fe0e015f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 --2- 172.21.15.90:0/1996802818 >> v2:172.21.15.102:3300/0 conn(0x7fe0e015bbb0 0x7fe0e015f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 -- 172.21.15.90:0/1996802818 >> v1:172.21.15.102:6789/0 conn(0x7fe0e00b4c20 legacy=0x7fe0e0161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.149+0000 7fe0d6000640 1 -- 172.21.15.90:0/1996802818 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e00b32f0 con 0x7fe0e015bf90 2024-03-31T15:54:56.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0d6000640 1 --2- 172.21.15.90:0/1996802818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e015bf90 0x7fe0e0071260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00b2870 tx=0x7fe0c801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f372d84023df0e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0de000640 1 -- 172.21.15.90:0/1996802818 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80ab660 con 0x7fe0e015bf90 2024-03-31T15:54:56.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0de000640 1 -- 172.21.15.90:0/1996802818 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c80b0070 con 0x7fe0e015bf90 2024-03-31T15:54:56.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0de000640 1 -- 172.21.15.90:0/1996802818 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80aba20 con 0x7fe0e015bf90 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 -- 172.21.15.90:0/1996802818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e015bf90 msgr2=0x7fe0e0071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/1996802818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e015bf90 0x7fe0e0071260 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00b2870 tx=0x7fe0c801c460 comp rx=0 tx=0).stop 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 -- 172.21.15.90:0/1996802818 shutdown_connections 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/1996802818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e015bf90 0x7fe0e0071260 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/1996802818 >> v2:172.21.15.102:3300/0 conn(0x7fe0e015bbb0 0x7fe0e015f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 -- 172.21.15.90:0/1996802818 >> 172.21.15.90:0/1996802818 conn(0x7fe0e00b18b0 msgr2=0x7fe0e00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 -- 172.21.15.90:0/1996802818 shutdown_connections 2024-03-31T15:54:56.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.153+0000 7fe0e5c00640 1 -- 172.21.15.90:0/1996802818 wait complete. 2024-03-31T15:54:56.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 Processor -- start 2024-03-31T15:54:56.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 -- start start 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 0x7fe0e00cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 0x7fe0e00ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e016b0f0 con 0x7fe0e00b4c20 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e0071bb0 con 0x7fe0e00d0280 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0e5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e016b270 con 0x7fe0e015bbb0 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41650/0 (socket says 172.21.15.90:41650) 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 -- 172.21.15.90:0/2407776629 learned_addr learned my addr 172.21.15.90:0/2407776629 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0d6000640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 0x7fe0e00cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0df400640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 0x7fe0e00ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 -- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 msgr2=0x7fe0e00cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 0x7fe0e00cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 -- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 msgr2=0x7fe0e00ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 0x7fe0e00ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0dea00640 1 -- 172.21.15.90:0/2407776629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e0155e60 con 0x7fe0e00b4c20 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0d6000640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 0x7fe0e00cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.157+0000 7fe0df400640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 0x7fe0e00ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:56.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0dea00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d830 tx=0x7fe0d000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d000b6c0 con 0x7fe0e00b4c20 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0d0019070 con 0x7fe0e00b4c20 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d000ba60 con 0x7fe0e00b4c20 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e009c6a0 con 0x7fe0e00b4c20 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e009cc20 con 0x7fe0e00b4c20 2024-03-31T15:54:56.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e00685f0 con 0x7fe0e00b4c20 2024-03-31T15:54:56.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe0d001e460 con 0x7fe0e00b4c20 2024-03-31T15:54:56.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 0x7fe0bc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe0d0087490 con 0x7fe0e00b4c20 2024-03-31T15:54:56.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.161+0000 7fe0d6000640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 0x7fe0bc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.165+0000 7fe0d6000640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 0x7fe0bc06a870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80af000 tx=0x7fe0c80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.165+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe0e0155e60 con 0x7fe0e00b4c20 2024-03-31T15:54:56.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.325+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe0e00733f0 con 0x7fe0e00b4c20 2024-03-31T15:54:56.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.325+0000 7fe0d7400640 1 -- 172.21.15.90:0/2407776629 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fe0d0057640 con 0x7fe0e00b4c20 2024-03-31T15:54:56.318 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:56.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.325+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 msgr2=0x7fe0bc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 0x7fe0bc06a870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80af000 tx=0x7fe0c80a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:56.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 msgr2=0x7fe0e0155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d830 tx=0x7fe0d000dcf0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 shutdown_connections 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe0bc0683b0 0x7fe0bc06a870 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0e00d0280 0x7fe0e00ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0e015bbb0 0x7fe0e00cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 --2- 172.21.15.90:0/2407776629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e00b4c20 0x7fe0e0155920 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 >> 172.21.15.90:0/2407776629 conn(0x7fe0e00b18b0 msgr2=0x7fe0e015c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 shutdown_connections 2024-03-31T15:54:56.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.329+0000 7fe0e5c00640 1 -- 172.21.15.90:0/2407776629 wait complete. 2024-03-31T15:54:56.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:56.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 Processor -- start 2024-03-31T15:54:56.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 -- start start 2024-03-31T15:54:56.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f251806d020 0x7f25180ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f251815e8c0 con 0x7f251806d460 2024-03-31T15:54:56.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25180d0ff0 con 0x7f251815e4e0 2024-03-31T15:54:56.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f251d800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25180d12d0 con 0x7f251806d020 2024-03-31T15:54:56.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2517400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f251806d460 0x7f25180cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50180/0 (socket says 172.21.15.90:50180) 2024-03-31T15:54:56.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2517400640 1 -- 172.21.15.90:0/4213403043 learned_addr learned my addr 172.21.15.90:0/4213403043 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f250fe00640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f251806d020 0x7f25180ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71540639 0 0) 0x7f251815e8c0 con 0x7f251806d460 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24fc003530 con 0x7f251806d460 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358689519 0 0) 0x7f25180d0ff0 con 0x7f251815e4e0 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f251815e8c0 con 0x7f251815e4e0 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 776082238 0 0) 0x7f24fc003530 con 0x7f251806d460 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f251806d020 msgr2=0x7f25180ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f251806d020 0x7f25180ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 >> v1:172.21.15.102:6789/0 conn(0x7f251815e4e0 legacy=0x7f251806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25180cc710 con 0x7f251806d460 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f2510001a00 con 0x7f251806d460 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 >> v1:172.21.15.90:6789/0 conn(0x7f251806d460 legacy=0x7f25180cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc003b80 0x7f24fc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24fc004810 0x7f24fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24fc0051d0 0x7f24fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25180d12d0 con 0x7f24fc0051d0 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc00c0a0 con 0x7f24fc004810 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f24fc00c2a0 con 0x7f24fc003b80 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f250fe00640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f251806d020 0x7f25180ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.485+0000 7f2517400640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24fc0051d0 0x7f24fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2516a00640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc003b80 0x7f24fc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f250fe00640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24fc004810 0x7f24fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 -- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc003b80 msgr2=0x7f24fc003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc003b80 0x7f24fc003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 -- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24fc004810 msgr2=0x7f24fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24fc004810 0x7f24fc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 -- 172.21.15.90:0/4213403043 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24fc003530 con 0x7f24fc0051d0 2024-03-31T15:54:56.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24fc0051d0 0x7f24fc00b9d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f251806d8f0 tx=0x7f251001d220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce05be427e0dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251000a390 con 0x7f24fc0051d0 2024-03-31T15:54:56.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25100ad430 con 0x7f24fc0051d0 2024-03-31T15:54:56.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2516000640 1 -- 172.21.15.90:0/4213403043 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100ad840 con 0x7f24fc0051d0 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f251d800640 1 -- 172.21.15.90:0/4213403043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24fc0051d0 msgr2=0x7f24fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f251d800640 1 --2- 172.21.15.90:0/4213403043 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f24fc0051d0 0x7f24fc00b9d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f251806d8f0 tx=0x7f251001d220 comp rx=0 tx=0).stop 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f2517400640 1 -- 172.21.15.90:0/4213403043 reap_dead start 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f251d800640 1 -- 172.21.15.90:0/4213403043 shutdown_connections 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.489+0000 7f251d800640 1 -- 172.21.15.90:0/4213403043 >> 172.21.15.90:0/4213403043 conn(0x7f25180d1f20 msgr2=0x7f25180d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.493+0000 7f251d800640 1 -- 172.21.15.90:0/4213403043 shutdown_connections 2024-03-31T15:54:56.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.493+0000 7f251d800640 1 -- 172.21.15.90:0/4213403043 wait complete. 2024-03-31T15:54:56.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.493+0000 7f251d800640 1 Processor -- start 2024-03-31T15:54:56.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.493+0000 7f251d800640 1 -- start start 2024-03-31T15:54:56.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 0x7f25180cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251815c260 0x7f251814e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25180cf7b0 con 0x7f251815c6e0 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25180d2d60 con 0x7f251815c260 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25180d0c10 con 0x7f24fc0051d0 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41672/0 (socket says 172.21.15.90:41672) 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 -- 172.21.15.90:0/3241559154 learned_addr learned my addr 172.21.15.90:0/3241559154 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:56.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f250fe00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251815c260 0x7f251814e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2516a00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 0x7f25180cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 -- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 msgr2=0x7f25180cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 0x7f25180cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 -- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251815c260 msgr2=0x7f251814e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251815c260 0x7f251814e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 -- 172.21.15.90:0/3241559154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2518160690 con 0x7f251815c6e0 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2516a00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 0x7f25180cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f2517400640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f251001d700 tx=0x7f24fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25100be070 con 0x7f251815c6e0 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25100b3040 con 0x7f251815c6e0 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251000a390 con 0x7f251815c6e0 2024-03-31T15:54:56.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251815ed80 con 0x7f251815c6e0 2024-03-31T15:54:56.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251815fdf0 con 0x7f251815c6e0 2024-03-31T15:54:56.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25180685f0 con 0x7f251815c6e0 2024-03-31T15:54:56.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.497+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25100ad430 con 0x7f251815c6e0 2024-03-31T15:54:56.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.501+0000 7f250ea00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 0x7f24f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.501+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2510121380 con 0x7f251815c6e0 2024-03-31T15:54:56.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.501+0000 7f2516a00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 0x7f24f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.501+0000 7f2516a00640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 0x7f24f406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2508005630 tx=0x7f2508009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.505+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2518160690 con 0x7f251815c6e0 2024-03-31T15:54:56.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.661+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f251815dbe0 con 0x7f251815c6e0 2024-03-31T15:54:56.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.661+0000 7f250ea00640 1 -- 172.21.15.90:0/3241559154 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f2510016f30 con 0x7f251815c6e0 2024-03-31T15:54:56.654 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:56.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.661+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 msgr2=0x7f24f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 0x7f24f406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2508005630 tx=0x7f2508009040 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 msgr2=0x7f251814e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f251001d700 tx=0x7f24fc005840 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 shutdown_connections 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f24f4068380 0x7f24f406a840 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f251815c6e0 0x7f251814e900 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f251815c260 0x7f251814e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 --2- 172.21.15.90:0/3241559154 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f24fc0051d0 0x7f25180cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 >> 172.21.15.90:0/3241559154 conn(0x7f25180d1f20 msgr2=0x7f2518070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 shutdown_connections 2024-03-31T15:54:56.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.665+0000 7f251d800640 1 -- 172.21.15.90:0/3241559154 wait complete. 2024-03-31T15:54:56.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.821+0000 7f209a800640 1 Processor -- start 2024-03-31T15:54:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 -- start start 2024-03-31T15:54:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20940cd900 con 0x7f20940cd470 2024-03-31T15:54:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f20940aed70 con 0x7f20940d2b20 2024-03-31T15:54:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f209a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f20940cefc0 con 0x7f209415c320 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f2093400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:54382/0 (socket says 172.21.15.90:54382) 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f2093400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:36970/0 (socket says 172.21.15.90:36970) 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 -- 172.21.15.90:0/362260107 learned_addr learned my addr 172.21.15.90:0/362260107 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f2093e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f20940cd470 0x7f20940cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56898/0 (socket says 172.21.15.90:56898) 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 -- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 msgr2=0x7f20940cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 -- 172.21.15.90:0/362260107 >> v1:172.21.15.102:6789/0 conn(0x7f20940cd470 legacy=0x7f20940cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f208be00640 1 -- 172.21.15.90:0/362260107 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20940cf2a0 con 0x7f20940d2b20 2024-03-31T15:54:56.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.825+0000 7f2093400640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f208be00640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940d2b20 0x7f20940ccf30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f207c009870 tx=0x7f207c01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c522da3851c116e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f2092a00640 1 -- 172.21.15.90:0/362260107 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0b2070 con 0x7f20940d2b20 2024-03-31T15:54:56.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f2092a00640 1 -- 172.21.15.90:0/362260107 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207c0ad680 con 0x7f20940d2b20 2024-03-31T15:54:56.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f2092a00640 1 -- 172.21.15.90:0/362260107 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c0ada50 con 0x7f20940d2b20 2024-03-31T15:54:56.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940d2b20 msgr2=0x7f20940ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940d2b20 0x7f20940ccf30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f207c009870 tx=0x7f207c01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- 172.21.15.90:0/362260107 shutdown_connections 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 --2- 172.21.15.90:0/362260107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- 172.21.15.90:0/362260107 >> 172.21.15.90:0/362260107 conn(0x7f20940d1f20 msgr2=0x7f20940d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- 172.21.15.90:0/362260107 shutdown_connections 2024-03-31T15:54:56.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- 172.21.15.90:0/362260107 wait complete. 2024-03-31T15:54:56.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 Processor -- start 2024-03-31T15:54:56.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.829+0000 7f209a800640 1 -- start start 2024-03-31T15:54:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940d2b20 0x7f209415c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f2094153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2094168ec0 con 0x7f20940d2b20 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f20940d0eb0 con 0x7f20940cd470 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2094169040 con 0x7f209415c320 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f208be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940d2b20 0x7f209415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:54390/0 (socket says 172.21.15.90:54390) 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 -- 172.21.15.90:0/503420420 learned_addr learned my addr 172.21.15.90:0/503420420 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093e00640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f2094153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 -- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 msgr2=0x7f2094153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f2094153b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 -- 172.21.15.90:0/503420420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940d2b20 msgr2=0x7f209415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940d2b20 0x7f209415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:56.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 -- 172.21.15.90:0/503420420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2094154700 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f2093400640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f208400e980 tx=0x7f208400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2084017070 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f208400ca00 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208400cdd0 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209415a5d0 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209415ab90 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.833+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2094166c80 con 0x7f20940cd470 2024-03-31T15:54:56.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.837+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f208401a070 con 0x7f20940cd470 2024-03-31T15:54:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.837+0000 7f208b400640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.837+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2084086cf0 con 0x7f20940cd470 2024-03-31T15:54:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.837+0000 7f208be00640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:56.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.837+0000 7f208be00640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f207c0af000 tx=0x7f207c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:56.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:56.841+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2094166c80 con 0x7f20940cd470 2024-03-31T15:54:57.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.013+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f20940685f0 con 0x7f20940cd470 2024-03-31T15:54:57.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.013+0000 7f208b400640 1 -- 172.21.15.90:0/503420420 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2094154700 con 0x7f20940cd470 2024-03-31T15:54:57.007 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 msgr2=0x7f207006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f207c0af000 tx=0x7f207c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 msgr2=0x7f2094166680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f208400e980 tx=0x7f208400ee40 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 shutdown_connections 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f209415c320 0x7f2094153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940d2b20 0x7f209415c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 --2- 172.21.15.90:0/503420420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20940cd470 0x7f2094166680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 >> 172.21.15.90:0/503420420 conn(0x7f20940d1f20 msgr2=0x7f2094070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:57.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 shutdown_connections 2024-03-31T15:54:57.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.017+0000 7f209a800640 1 -- 172.21.15.90:0/503420420 wait complete. 2024-03-31T15:54:57.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:57.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.173+0000 7f1cf2600640 1 Processor -- start 2024-03-31T15:54:57.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.173+0000 7f1cf2600640 1 -- start start 2024-03-31T15:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf2600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1cec06fc40 0x7f1cec0cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf2600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec0d0590 con 0x7f1cec06bd00 2024-03-31T15:54:57.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf2600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cec0d0870 con 0x7f1cec1456c0 2024-03-31T15:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf2600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1cec06e7a0 con 0x7f1cec06fc40 2024-03-31T15:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cebe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1cec06bd00 0x7f1cec0cf8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50196/0 (socket says 172.21.15.90:50196) 2024-03-31T15:54:57.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cebe00640 1 -- 172.21.15.90:0/1481710462 learned_addr learned my addr 172.21.15.90:0/1481710462 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827949739 0 0) 0x7f1cec0d0590 con 0x7f1cec06bd00 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd4003530 con 0x7f1cec06bd00 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1c00640 1 --2- 172.21.15.90:0/1481710462 >> v2:172.21.15.102:3300/0 conn(0x7f1cec06fc40 0x7f1cec0cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1015932505 0 0) 0x7f1cd4003530 con 0x7f1cec06bd00 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 >> v2:172.21.15.102:3300/0 conn(0x7f1cec06fc40 msgr2=0x7f1cec0cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 --2- 172.21.15.90:0/1481710462 >> v2:172.21.15.102:3300/0 conn(0x7f1cec06fc40 0x7f1cec0cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 >> v1:172.21.15.102:6789/0 conn(0x7f1cec1456c0 legacy=0x7f1cec06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec06e150 con 0x7f1cec06bd00 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f1cd8001680 con 0x7f1cec06bd00 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1c00640 1 --2- 172.21.15.90:0/1481710462 >> v2:172.21.15.102:3300/0 conn(0x7f1cec06fc40 0x7f1cec0cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 >> v1:172.21.15.90:6789/0 conn(0x7f1cec06bd00 legacy=0x7f1cec0cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cd4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cd4004810 0x7f1cd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cd40052f0 0x7f1cd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec06e7a0 con 0x7f1cd40039c0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd400c1c0 con 0x7f1cd40052f0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd400c3c0 con 0x7f1cd4004810 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cd4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cebe00640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cd4004810 0x7f1cd4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1c00640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cd40052f0 0x7f1cd400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 -- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cd4004810 msgr2=0x7f1cd4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cd4004810 0x7f1cd4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 -- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cd40052f0 msgr2=0x7f1cd400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cd40052f0 0x7f1cd400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1200640 1 -- 172.21.15.90:0/1481710462 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd4003530 con 0x7f1cd40039c0 2024-03-31T15:54:57.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.177+0000 7f1cf1c00640 1 -- 172.21.15.90:0/1481710462 reap_dead start 2024-03-31T15:54:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf1200640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cd4003de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0001750 tx=0x7f1ce001c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d914174e87bf5759 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce00013a0 con 0x7f1cd40039c0 2024-03-31T15:54:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce00052f0 con 0x7f1cd40039c0 2024-03-31T15:54:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1ceb400640 1 -- 172.21.15.90:0/1481710462 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0003840 con 0x7f1cd40039c0 2024-03-31T15:54:57.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 -- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 msgr2=0x7f1cd4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cd4003de0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0001750 tx=0x7f1ce001c390 comp rx=0 tx=0).stop 2024-03-31T15:54:57.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 -- 172.21.15.90:0/1481710462 shutdown_connections 2024-03-31T15:54:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 --2- 172.21.15.90:0/1481710462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cd4003de0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 -- 172.21.15.90:0/1481710462 >> 172.21.15.90:0/1481710462 conn(0x7f1cec0d0c60 msgr2=0x7f1cec0d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 -- 172.21.15.90:0/1481710462 shutdown_connections 2024-03-31T15:54:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 -- 172.21.15.90:0/1481710462 wait complete. 2024-03-31T15:54:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.181+0000 7f1cf2600640 1 Processor -- start 2024-03-31T15:54:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- start start 2024-03-31T15:54:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cec143bb0 0x7f1cec144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec15e8b0 0x7f1cec15ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec0d0230 con 0x7f1cd40039c0 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1cec0d00b0 con 0x7f1cec15e8b0 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1cec0721f0 con 0x7f1cec143bb0 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41714/0 (socket says 172.21.15.90:41714) 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 -- 172.21.15.90:0/3233314968 learned_addr learned my addr 172.21.15.90:0/3233314968 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cebe00640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cec143bb0 0x7f1cec144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1c00640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec15e8b0 0x7f1cec15ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 -- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cec143bb0 msgr2=0x7f1cec144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cec143bb0 0x7f1cec144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 -- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec15e8b0 msgr2=0x7f1cec15ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec15e8b0 0x7f1cec15ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 -- 172.21.15.90:0/3233314968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cec070020 con 0x7f1cd40039c0 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf1200640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0005040 tx=0x7f1cd4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0007a40 con 0x7f1cd40039c0 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce00bb070 con 0x7f1cd40039c0 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce00013a0 con 0x7f1cd40039c0 2024-03-31T15:54:57.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cec13b340 con 0x7f1cd40039c0 2024-03-31T15:54:57.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cec1f7810 con 0x7f1cd40039c0 2024-03-31T15:54:57.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.185+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cec068060 con 0x7f1cd40039c0 2024-03-31T15:54:57.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.189+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1ce00b4040 con 0x7f1cd40039c0 2024-03-31T15:54:57.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.189+0000 7f1ce8c00640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 0x7f1ccc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.189+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1ce0120bd0 con 0x7f1cd40039c0 2024-03-31T15:54:57.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.189+0000 7f1cebe00640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 0x7f1ccc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.189+0000 7f1cebe00640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 0x7f1ccc06a840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06f010 tx=0x7f1cd8000f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.193+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1ce00160d0 con 0x7f1cd40039c0 2024-03-31T15:54:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.349+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1cec068d00 con 0x7f1cd40039c0 2024-03-31T15:54:57.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1ce8c00640 1 -- 172.21.15.90:0/3233314968 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f1ce00f06a0 con 0x7f1cd40039c0 2024-03-31T15:54:57.344 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 msgr2=0x7f1ccc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 0x7f1ccc06a840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1cec06f010 tx=0x7f1cd8000f90 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 msgr2=0x7f1cec13ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0005040 tx=0x7f1cd4005840 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 shutdown_connections 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ccc068380 0x7f1ccc06a840 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cec15e8b0 0x7f1cec15ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1cec143bb0 0x7f1cec144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 --2- 172.21.15.90:0/3233314968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1cd40039c0 0x7f1cec13ae00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 >> 172.21.15.90:0/3233314968 conn(0x7f1cec0d0c60 msgr2=0x7f1cec0c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 shutdown_connections 2024-03-31T15:54:57.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.353+0000 7f1cf2600640 1 -- 172.21.15.90:0/3233314968 wait complete. 2024-03-31T15:54:57.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:57.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 Processor -- start 2024-03-31T15:54:57.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 -- start start 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 0x7fb0440d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb0440d2940 0x7fb044161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0441622d0 0x7fb04406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb04415e700 con 0x7fb0440d2940 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb0440d2df0 con 0x7fb04415e320 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.509+0000 7fb049a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440d1400 con 0x7fb0441622d0 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb0440d2940 0x7fb044161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb0440d2940 0x7fb044161d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41726/0 (socket says 172.21.15.90:41726) 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 -- 172.21.15.90:0/1986066543 learned_addr learned my addr 172.21.15.90:0/1986066543 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042a00640 1 --2- 172.21.15.90:0/1986066543 >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 0x7fb0440d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb043400640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0441622d0 0x7fb04406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 -- 172.21.15.90:0/1986066543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0441622d0 msgr2=0x7fb04406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0441622d0 0x7fb04406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 -- 172.21.15.90:0/1986066543 >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 msgr2=0x7fb0440d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 --2- 172.21.15.90:0/1986066543 >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 0x7fb0440d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 -- 172.21.15.90:0/1986066543 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0440b1a20 con 0x7fb0440d2940 2024-03-31T15:54:57.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042a00640 1 --2- 172.21.15.90:0/1986066543 >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 0x7fb0440d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:57.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb042000640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0440d2940 0x7fb044161d90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb030009870 tx=0x7fb03001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9481d77b5f05c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb041600640 1 -- 172.21.15.90:0/1986066543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0300b0070 con 0x7fb0440d2940 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb041600640 1 -- 172.21.15.90:0/1986066543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0300ab600 con 0x7fb0440d2940 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.513+0000 7fb041600640 1 -- 172.21.15.90:0/1986066543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0300ab9d0 con 0x7fb0440d2940 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- 172.21.15.90:0/1986066543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0440d2940 msgr2=0x7fb044161d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0440d2940 0x7fb044161d90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb030009870 tx=0x7fb03001c260 comp rx=0 tx=0).stop 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- 172.21.15.90:0/1986066543 shutdown_connections 2024-03-31T15:54:57.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0441622d0 0x7fb04406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- 172.21.15.90:0/1986066543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0440d2940 0x7fb044161d90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- 172.21.15.90:0/1986066543 >> v2:172.21.15.102:3300/0 conn(0x7fb04415e320 0x7fb0440d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- 172.21.15.90:0/1986066543 >> 172.21.15.90:0/1986066543 conn(0x7fb04408bfd0 msgr2=0x7fb0440d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- 172.21.15.90:0/1986066543 shutdown_connections 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- 172.21.15.90:0/1986066543 wait complete. 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 Processor -- start 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- start start 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0440d2940 0x7fb044150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb04415e320 0x7fb044154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb0440b2890 con 0x7fb0441622d0 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0440b2710 con 0x7fb04415e320 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb049a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb0440b2a10 con 0x7fb0440d2940 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41736/0 (socket says 172.21.15.90:41736) 2024-03-31T15:54:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb042000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb04415e320 0x7fb044154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 -- 172.21.15.90:0/3892598163 learned_addr learned my addr 172.21.15.90:0/3892598163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:57.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb042a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0440d2940 0x7fb044150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 -- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0440d2940 msgr2=0x7fb044150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0440d2940 0x7fb044150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 -- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb04415e320 msgr2=0x7fb044154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.517+0000 7fb043400640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb04415e320 0x7fb044154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb043400640 1 -- 172.21.15.90:0/3892598163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb044151e30 con 0x7fb0441622d0 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb043400640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009d30 tx=0x7fb03c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c018020 con 0x7fb0441622d0 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c014480 con 0x7fb0441622d0 2024-03-31T15:54:57.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c014870 con 0x7fb0441622d0 2024-03-31T15:54:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb044152110 con 0x7fb0441622d0 2024-03-31T15:54:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0440c6c50 con 0x7fb0441622d0 2024-03-31T15:54:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb044068660 con 0x7fb0441622d0 2024-03-31T15:54:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.521+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb03c01b020 con 0x7fb0441622d0 2024-03-31T15:54:57.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.525+0000 7fb03aa00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 0x7fb02406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.525+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb03c087730 con 0x7fb0441622d0 2024-03-31T15:54:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.525+0000 7fb042a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 0x7fb02406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.525+0000 7fb042a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 0x7fb02406a840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb044156260 tx=0x7fb034008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:57.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.533+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb03c022070 con 0x7fb0441622d0 2024-03-31T15:54:57.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.701+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb044157ee0 con 0x7fb0441622d0 2024-03-31T15:54:57.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.701+0000 7fb03aa00640 1 -- 172.21.15.90:0/3892598163 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fb03c010040 con 0x7fb0441622d0 2024-03-31T15:54:57.694 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 msgr2=0x7fb02406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 0x7fb02406a840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb044156260 tx=0x7fb034008040 comp rx=0 tx=0).stop 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 msgr2=0x7fb0441552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009d30 tx=0x7fb03c00c490 comp rx=0 tx=0).stop 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 shutdown_connections 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb024068380 0x7fb02406a840 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0441622d0 0x7fb0441552b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb04415e320 0x7fb044154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 --2- 172.21.15.90:0/3892598163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb0440d2940 0x7fb044150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 >> 172.21.15.90:0/3892598163 conn(0x7fb04408bfd0 msgr2=0x7fb0440b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:57.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 shutdown_connections 2024-03-31T15:54:57.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:57.705+0000 7fb049a00640 1 -- 172.21.15.90:0/3892598163 wait complete. 2024-03-31T15:54:57.722 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:54:57.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:57.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-31T15:54:57.743 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:54:57.744 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:54:57.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:54:57.744 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:54:57.796 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:57.796 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:57.805 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:57.805 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:57.857 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:54:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.001+0000 7f973fe00640 1 Processor -- start 2024-03-31T15:54:57.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.001+0000 7f973fe00640 1 -- start start 2024-03-31T15:54:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.001+0000 7f973fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.001+0000 7f973fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 0x7f974006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9740067330 con 0x7f97400cc7e0 2024-03-31T15:54:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f974006d9e0 con 0x7f97400af540 2024-03-31T15:54:57.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f974006dc30 con 0x7f974015af00 2024-03-31T15:54:57.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 0x7f974006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:57.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f97400cc7e0 0x7f97400ad920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56910/0 (socket says 172.21.15.90:56910) 2024-03-31T15:54:57.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 0x7f974006e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:54442/0 (socket says 172.21.15.90:54442) 2024-03-31T15:54:57.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973f400640 1 -- 172.21.15.90:0/2074392107 learned_addr learned my addr 172.21.15.90:0/2074392107 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:37020/0 (socket says 172.21.15.90:37020) 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973d600640 1 -- 172.21.15.90:0/2074392107 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061117733 0 0) 0x7f9740067330 con 0x7f97400cc7e0 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 -- 172.21.15.90:0/2074392107 >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 msgr2=0x7f974006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 --2- 172.21.15.90:0/2074392107 >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 0x7f974006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 -- 172.21.15.90:0/2074392107 >> v1:172.21.15.102:6789/0 conn(0x7f97400cc7e0 legacy=0x7f97400ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:57.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.005+0000 7f973ea00640 1 -- 172.21.15.90:0/2074392107 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97400675d0 con 0x7f974015af00 2024-03-31T15:54:58.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973ea00640 1 --2- 172.21.15.90:0/2074392107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f97300097c0 tx=0x7f973001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9142fde2c54aed2a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973d600640 1 -- 172.21.15.90:0/2074392107 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300b0070 con 0x7f974015af00 2024-03-31T15:54:58.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973d600640 1 -- 172.21.15.90:0/2074392107 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97300ab600 con 0x7f974015af00 2024-03-31T15:54:58.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973d600640 1 -- 172.21.15.90:0/2074392107 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300ab9d0 con 0x7f974015af00 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- 172.21.15.90:0/2074392107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 msgr2=0x7f974006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- 172.21.15.90:0/2074392107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f97300097c0 tx=0x7f973001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- 172.21.15.90:0/2074392107 shutdown_connections 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- 172.21.15.90:0/2074392107 >> v2:172.21.15.102:3300/0 conn(0x7f97400af540 0x7f974006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- 172.21.15.90:0/2074392107 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f974015af00 0x7f974006e3f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- 172.21.15.90:0/2074392107 >> 172.21.15.90:0/2074392107 conn(0x7f97400b5130 msgr2=0x7f97400b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- 172.21.15.90:0/2074392107 shutdown_connections 2024-03-31T15:54:58.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- 172.21.15.90:0/2074392107 wait complete. 2024-03-31T15:54:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 Processor -- start 2024-03-31T15:54:58.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- start start 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97400af540 0x7f97400caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f974015af00 0x7f9740149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97400730c0 con 0x7f97400cc7e0 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9740072f40 con 0x7f974015af00 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.009+0000 7f973fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9740073240 con 0x7f97400af540 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41744/0 (socket says 172.21.15.90:41744) 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 -- 172.21.15.90:0/1793013358 learned_addr learned my addr 172.21.15.90:0/1793013358 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973ea00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97400af540 0x7f97400caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973f400640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f974015af00 0x7f9740149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 -- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97400af540 msgr2=0x7f97400caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97400af540 0x7f97400caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 -- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f974015af00 msgr2=0x7f9740149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f974015af00 0x7f9740149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 -- 172.21.15.90:0/1793013358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97400cbf30 con 0x7f97400cc7e0 2024-03-31T15:54:58.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973e000640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ed40 tx=0x7f972c00d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c00de30 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f972c010650 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97400cbc50 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972c010a00 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f974014bd20 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9740068d00 con 0x7f97400cc7e0 2024-03-31T15:54:58.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.013+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f972c010ba0 con 0x7f97400cc7e0 2024-03-31T15:54:58.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.017+0000 7f9736a00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 0x7f972006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.017+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f972c087700 con 0x7f97400cc7e0 2024-03-31T15:54:58.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.017+0000 7f973ea00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 0x7f972006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.017+0000 7f973ea00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 0x7f972006a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97300ad000 tx=0x7f97300a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.017+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f972c014030 con 0x7f97400cc7e0 2024-03-31T15:54:58.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.181+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97401496e0 con 0x7f97400cc7e0 2024-03-31T15:54:58.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.181+0000 7f9736a00640 1 -- 172.21.15.90:0/1793013358 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f972c057260 con 0x7f97400cc7e0 2024-03-31T15:54:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:54:58.176 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:54:58.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 msgr2=0x7f972006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 0x7f972006a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f97300ad000 tx=0x7f97300a6000 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 msgr2=0x7f97400cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ed40 tx=0x7f972c00d720 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 shutdown_connections 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9720068380 0x7f972006a840 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f974015af00 0x7f9740149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97400cc7e0 0x7f97400cb410 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 --2- 172.21.15.90:0/1793013358 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97400af540 0x7f97400caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 >> 172.21.15.90:0/1793013358 conn(0x7f97400b5130 msgr2=0x7f97400b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 shutdown_connections 2024-03-31T15:54:58.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.185+0000 7f973fe00640 1 -- 172.21.15.90:0/1793013358 wait complete. 2024-03-31T15:54:58.197 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:54:58.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:54:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.341+0000 7f7486000640 1 Processor -- start 2024-03-31T15:54:58.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f7486000640 1 -- start start 2024-03-31T15:54:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f7486000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f748015b040 0x7f748015b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f7486000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7480072a60 con 0x7f74800725a0 2024-03-31T15:54:58.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f7486000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74800aeac0 con 0x7f7480071940 2024-03-31T15:54:58.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f7486000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f74800ae890 con 0x7f748015b040 2024-03-31T15:54:58.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f74800725a0 0x7f74800b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50204/0 (socket says 172.21.15.90:50204) 2024-03-31T15:54:58.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747fe00640 1 -- 172.21.15.90:0/1275883888 learned_addr learned my addr 172.21.15.90:0/1275883888 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:58.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747f400640 1 --2- 172.21.15.90:0/1275883888 >> v2:172.21.15.102:3300/0 conn(0x7f748015b040 0x7f748015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104614589 0 0) 0x7f74800aeac0 con 0x7f7480071940 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7464003530 con 0x7f7480071940 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546313293 0 0) 0x7f7480072a60 con 0x7f74800725a0 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74800aeac0 con 0x7f74800725a0 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747f400640 1 -- 172.21.15.90:0/1275883888 >> v1:172.21.15.102:6789/0 conn(0x7f7480071940 legacy=0x7f7480071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747f400640 1 -- 172.21.15.90:0/1275883888 >> v1:172.21.15.90:6789/0 conn(0x7f74800725a0 legacy=0x7f74800b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.345+0000 7f747f400640 1 -- 172.21.15.90:0/1275883888 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74800aeda0 con 0x7f748015b040 2024-03-31T15:54:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f747f400640 1 --2- 172.21.15.90:0/1275883888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f748015b040 0x7f748015b420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f74700097c0 tx=0x7f747001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f0709e19274fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74700b0070 con 0x7f748015b040 2024-03-31T15:54:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74700ab600 con 0x7f748015b040 2024-03-31T15:54:58.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f747ea00640 1 -- 172.21.15.90:0/1275883888 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74700ab9d0 con 0x7f748015b040 2024-03-31T15:54:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 -- 172.21.15.90:0/1275883888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f748015b040 msgr2=0x7f748015b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 --2- 172.21.15.90:0/1275883888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f748015b040 0x7f748015b420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f74700097c0 tx=0x7f747001c410 comp rx=0 tx=0).stop 2024-03-31T15:54:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 -- 172.21.15.90:0/1275883888 shutdown_connections 2024-03-31T15:54:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 --2- 172.21.15.90:0/1275883888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f748015b040 0x7f748015b420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 -- 172.21.15.90:0/1275883888 >> 172.21.15.90:0/1275883888 conn(0x7f74800b18b0 msgr2=0x7f74800b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 -- 172.21.15.90:0/1275883888 shutdown_connections 2024-03-31T15:54:58.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 -- 172.21.15.90:0/1275883888 wait complete. 2024-03-31T15:54:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.349+0000 7f7486000640 1 Processor -- start 2024-03-31T15:54:58.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- start start 2024-03-31T15:54:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 0x7f748013a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74800725a0 0x7f748013b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7480168e40 con 0x7f748015b040 2024-03-31T15:54:58.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f74800b35c0 con 0x7f74800725a0 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7480168fc0 con 0x7f7480071940 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41746/0 (socket says 172.21.15.90:41746) 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 -- 172.21.15.90:0/1298414443 learned_addr learned my addr 172.21.15.90:0/1298414443 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747f400640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 0x7f748013a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7477e00640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74800725a0 0x7f748013b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 -- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 msgr2=0x7f748013a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 0x7f748013a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 -- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74800725a0 msgr2=0x7f748013b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74800725a0 0x7f748013b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 -- 172.21.15.90:0/1298414443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f748014cc80 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747f400640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 0x7f748013a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f747fe00640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f747800caa0 tx=0x7f747800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478017070 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f747800ac00 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f748009bfa0 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478012400 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f748009c560 con 0x7f748015b040 2024-03-31T15:54:58.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.353+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74800685f0 con 0x7f748015b040 2024-03-31T15:54:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.357+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f747801a070 con 0x7f748015b040 2024-03-31T15:54:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.357+0000 7f7477400640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 0x7f745c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.357+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7478086a90 con 0x7f748015b040 2024-03-31T15:54:58.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.357+0000 7f747f400640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 0x7f745c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.357+0000 7f747f400640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 0x7f745c06a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74700ad000 tx=0x7f7470008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.365+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f74780565f0 con 0x7f748015b040 2024-03-31T15:54:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.521+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74800ae670 con 0x7f748015b040 2024-03-31T15:54:58.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.521+0000 7f7477400640 1 -- 172.21.15.90:0/1298414443 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f747805a8f0 con 0x7f748015b040 2024-03-31T15:54:58.515 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 msgr2=0x7f745c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 0x7f745c06a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f74700ad000 tx=0x7f7470008040 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 msgr2=0x7f748014c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f747800caa0 tx=0x7f747800cf60 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 shutdown_connections 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f745c068380 0x7f745c06a840 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f748015b040 0x7f748014c1c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74800725a0 0x7f748013b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 --2- 172.21.15.90:0/1298414443 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7480071940 0x7f748013a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 >> 172.21.15.90:0/1298414443 conn(0x7f74800b18b0 msgr2=0x7f7480150f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 shutdown_connections 2024-03-31T15:54:58.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.525+0000 7f7486000640 1 -- 172.21.15.90:0/1298414443 wait complete. 2024-03-31T15:54:58.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:54:58.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 Processor -- start 2024-03-31T15:54:58.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 -- start start 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f31f815c320 0x7f31f80cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31f80af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f80cefc0 con 0x7f31f80d2b20 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f31f80aed70 con 0x7f31f815c320 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31fde00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f31f80cf2a0 con 0x7f31f80cd640 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f31f80d2b20 0x7f31f80ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50216/0 (socket says 172.21.15.90:50216) 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f6000640 1 -- 172.21.15.90:0/4008975524 learned_addr learned my addr 172.21.15.90:0/4008975524 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f6a00640 1 --2- 172.21.15.90:0/4008975524 >> v2:172.21.15.102:3300/0 conn(0x7f31f815c320 0x7f31f80cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31f80af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097542265 0 0) 0x7f31f80cefc0 con 0x7f31f80d2b20 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31dc003530 con 0x7f31f80d2b20 2024-03-31T15:54:58.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4247081520 0 0) 0x7f31dc003530 con 0x7f31f80d2b20 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 msgr2=0x7f31f80af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31f80af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 >> v2:172.21.15.102:3300/0 conn(0x7f31f815c320 msgr2=0x7f31f80cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 --2- 172.21.15.90:0/4008975524 >> v2:172.21.15.102:3300/0 conn(0x7f31f815c320 0x7f31f80cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f80cf580 con 0x7f31f80d2b20 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f31e4001680 con 0x7f31f80d2b20 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 >> v1:172.21.15.90:6789/0 conn(0x7f31f80d2b20 legacy=0x7f31f80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31dc0039c0 0x7f31dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80aed70 con 0x7f31dc004d50 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80cf2a0 con 0x7f31dc0039c0 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc00ba00 con 0x7f31f80cd640 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f6a00640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31dc0039c0 0x7f31dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f6000640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 -- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 msgr2=0x7f31dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f80cd640 0x7f31dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 -- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31dc0039c0 msgr2=0x7f31dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31dc0039c0 0x7f31dc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.685+0000 7f31f7400640 1 -- 172.21.15.90:0/4008975524 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc003530 con 0x7f31dc004d50 2024-03-31T15:54:58.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31f7400640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31dc0051d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009f90 tx=0x7f31f001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96a8f60802f94f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f00b0070 con 0x7f31dc004d50 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f80d0f20 con 0x7f31dc004d50 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31f5600640 1 -- 172.21.15.90:0/4008975524 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f0009c10 con 0x7f31dc004d50 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 -- 172.21.15.90:0/4008975524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 msgr2=0x7f31dc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 --2- 172.21.15.90:0/4008975524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31dc0051d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009f90 tx=0x7f31f001c920 comp rx=0 tx=0).stop 2024-03-31T15:54:58.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31f7400640 1 -- 172.21.15.90:0/4008975524 reap_dead start 2024-03-31T15:54:58.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 -- 172.21.15.90:0/4008975524 shutdown_connections 2024-03-31T15:54:58.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 -- 172.21.15.90:0/4008975524 >> 172.21.15.90:0/4008975524 conn(0x7f31f80d1f20 msgr2=0x7f31f80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 -- 172.21.15.90:0/4008975524 shutdown_connections 2024-03-31T15:54:58.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.693+0000 7f31fde00640 1 -- 172.21.15.90:0/4008975524 wait complete. 2024-03-31T15:54:58.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 Processor -- start 2024-03-31T15:54:58.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- start start 2024-03-31T15:54:58.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f815f040 0x7f31f8159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 0x7f31f80b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80afd90 con 0x7f31dc004d50 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31f80afc10 con 0x7f31f80b5b30 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f31f80aff10 con 0x7f31f815f040 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41778/0 (socket says 172.21.15.90:41778) 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 -- 172.21.15.90:0/3138763988 learned_addr learned my addr 172.21.15.90:0/3138763988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f7400640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 0x7f31f80b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6000640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f815f040 0x7f31f8159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 -- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f815f040 msgr2=0x7f31f8159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f815f040 0x7f31f8159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 -- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 msgr2=0x7f31f80b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 0x7f31f80b5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 -- 172.21.15.90:0/3138763988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f815c970 con 0x7f31dc004d50 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f7400640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 0x7f31f80b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:54:58.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31f6a00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31e8011a00 tx=0x7f31e8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e800fb50 con 0x7f31dc004d50 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e800fcf0 con 0x7f31dc004d50 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f81f4610 con 0x7f31dc004d50 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e800fea0 con 0x7f31dc004d50 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.697+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f81f4bd0 con 0x7f31dc004d50 2024-03-31T15:54:58.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.701+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31f80685f0 con 0x7f31dc004d50 2024-03-31T15:54:58.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.701+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f31e8012040 con 0x7f31dc004d50 2024-03-31T15:54:58.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.701+0000 7f31ef400640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 0x7f31d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:58.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.701+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f31e8087550 con 0x7f31dc004d50 2024-03-31T15:54:58.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.701+0000 7f31f6000640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 0x7f31d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:58.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.705+0000 7f31f6000640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 0x7f31d806a870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f31e4000c10 tx=0x7f31e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.705+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f31e8019030 con 0x7f31dc004d50 2024-03-31T15:54:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.865+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f31f80c8a80 con 0x7f31dc004d50 2024-03-31T15:54:58.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.865+0000 7f31ef400640 1 -- 172.21.15.90:0/3138763988 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f31e80576a0 con 0x7f31dc004d50 2024-03-31T15:54:58.859 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 msgr2=0x7f31d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 0x7f31d806a870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f31e4000c10 tx=0x7f31e4009040 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 msgr2=0x7f31f81f3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f31e8011a00 tx=0x7f31e8011ec0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 shutdown_connections 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f31d80683b0 0x7f31d806a870 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31f80b5b30 0x7f31f80b5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f31f815f040 0x7f31f8159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 --2- 172.21.15.90:0/3138763988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f31dc004d50 0x7f31f81f3dc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 >> 172.21.15.90:0/3138763988 conn(0x7f31f80d1f20 msgr2=0x7f31f80d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 shutdown_connections 2024-03-31T15:54:58.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:58.869+0000 7f31fde00640 1 -- 172.21.15.90:0/3138763988 wait complete. 2024-03-31T15:54:58.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:54:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 Processor -- start 2024-03-31T15:54:59.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 -- start start 2024-03-31T15:54:59.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fde7815c320 0x7fde780cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde780cd640 0x7fde780af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde780cefc0 con 0x7fde780d2b20 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fde780aed70 con 0x7fde780cd640 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde7de00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fde780cf2a0 con 0x7fde7815c320 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde780cd640 0x7fde780af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fde780cd640 0x7fde780af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41782/0 (socket says 172.21.15.90:41782) 2024-03-31T15:54:59.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 -- 172.21.15.90:0/3284099379 learned_addr learned my addr 172.21.15.90:0/3284099379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde76a00640 1 --2- 172.21.15.90:0/3284099379 >> v2:172.21.15.102:3300/0 conn(0x7fde7815c320 0x7fde780cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 -- 172.21.15.90:0/3284099379 >> v2:172.21.15.102:3300/0 conn(0x7fde7815c320 msgr2=0x7fde780cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 --2- 172.21.15.90:0/3284099379 >> v2:172.21.15.102:3300/0 conn(0x7fde7815c320 0x7fde780cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 -- 172.21.15.90:0/3284099379 >> v1:172.21.15.90:6789/0 conn(0x7fde780d2b20 legacy=0x7fde780ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.037+0000 7fde77400640 1 -- 172.21.15.90:0/3284099379 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde780cf580 con 0x7fde780cd640 2024-03-31T15:54:59.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde77400640 1 --2- 172.21.15.90:0/3284099379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780cd640 0x7fde780af200 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fde780739b0 tx=0x7fde7001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366a7858366023b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde76000640 1 -- 172.21.15.90:0/3284099379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde700a5730 con 0x7fde780cd640 2024-03-31T15:54:59.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde76000640 1 -- 172.21.15.90:0/3284099379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde780d0f20 con 0x7fde780cd640 2024-03-31T15:54:59.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde76000640 1 -- 172.21.15.90:0/3284099379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde700a5af0 con 0x7fde780cd640 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 -- 172.21.15.90:0/3284099379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780cd640 msgr2=0x7fde780af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 --2- 172.21.15.90:0/3284099379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780cd640 0x7fde780af200 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fde780739b0 tx=0x7fde7001c960 comp rx=0 tx=0).stop 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 -- 172.21.15.90:0/3284099379 shutdown_connections 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 --2- 172.21.15.90:0/3284099379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780cd640 0x7fde780af200 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 --2- 172.21.15.90:0/3284099379 >> v2:172.21.15.102:3300/0 conn(0x7fde7815c320 0x7fde780cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 -- 172.21.15.90:0/3284099379 >> 172.21.15.90:0/3284099379 conn(0x7fde780d1f20 msgr2=0x7fde780d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 -- 172.21.15.90:0/3284099379 shutdown_connections 2024-03-31T15:54:59.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.041+0000 7fde7de00640 1 -- 172.21.15.90:0/3284099379 wait complete. 2024-03-31T15:54:59.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 Processor -- start 2024-03-31T15:54:59.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- start start 2024-03-31T15:54:59.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde780cd640 0x7fde7814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde7815c320 0x7fde7815a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fde780b02e0 con 0x7fde780d2b20 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde780b0160 con 0x7fde7815c320 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fde780b0460 con 0x7fde780cd640 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41794/0 (socket says 172.21.15.90:41794) 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 -- 172.21.15.90:0/119665755 learned_addr learned my addr 172.21.15.90:0/119665755 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde76a00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde780cd640 0x7fde7814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde77400640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde7815c320 0x7fde7815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 -- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde780cd640 msgr2=0x7fde7814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde780cd640 0x7fde7814b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 -- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde7815c320 msgr2=0x7fde7815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde7815c320 0x7fde7815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 -- 172.21.15.90:0/119665755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7806ca70 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6fe00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c370 tx=0x7fde6000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde600052d0 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde60005470 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60007260 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7806cd50 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7806d310 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.045+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde780685f0 con 0x7fde780d2b20 2024-03-31T15:54:59.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.049+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fde60007400 con 0x7fde780d2b20 2024-03-31T15:54:59.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.049+0000 7fde6f400640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.049+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fde60086240 con 0x7fde780d2b20 2024-03-31T15:54:59.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.049+0000 7fde76a00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.049+0000 7fde76a00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 0x7fde5406a840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde68007a60 tx=0x7fde68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.053+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fde60055d10 con 0x7fde780d2b20 2024-03-31T15:54:59.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde780cec80 con 0x7fde780d2b20 2024-03-31T15:54:59.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde6f400640 1 -- 172.21.15.90:0/119665755 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fde6005a010 con 0x7fde780d2b20 2024-03-31T15:54:59.206 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 msgr2=0x7fde5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 0x7fde5406a840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde68007a60 tx=0x7fde68008040 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 msgr2=0x7fde7806db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c370 tx=0x7fde6000c830 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 shutdown_connections 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fde54068380 0x7fde5406a840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde7815c320 0x7fde7815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fde780d2b20 0x7fde7806db10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 --2- 172.21.15.90:0/119665755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fde780cd640 0x7fde7814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 >> 172.21.15.90:0/119665755 conn(0x7fde780d1f20 msgr2=0x7fde7815d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 shutdown_connections 2024-03-31T15:54:59.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.213+0000 7fde7de00640 1 -- 172.21.15.90:0/119665755 wait complete. 2024-03-31T15:54:59.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:54:59.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 Processor -- start 2024-03-31T15:54:59.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 -- start start 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384806b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38480cffb0 0x7f384806ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38480d0590 con 0x7f384806bb30 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f384806bf90 con 0x7f38480cffb0 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f384806e7a0 con 0x7f38481456c0 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384d400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38480cffb0 0x7f384806ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f383fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f384806bb30 0x7f38480cf8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50230/0 (socket says 172.21.15.90:50230) 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f383fe00640 1 -- 172.21.15.90:0/1910201792 learned_addr learned my addr 172.21.15.90:0/1910201792 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384ca00640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384806b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384d400640 1 -- 172.21.15.90:0/1910201792 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 msgr2=0x7f384806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:59.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384d400640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384806b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384d400640 1 -- 172.21.15.90:0/1910201792 >> v1:172.21.15.90:6789/0 conn(0x7f384806bb30 legacy=0x7f38480cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f384d400640 1 -- 172.21.15.90:0/1910201792 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38480d0870 con 0x7f38480cffb0 2024-03-31T15:54:59.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.373+0000 7f3847e00640 1 -- 172.21.15.90:0/1910201792 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123282395 0 0) 0x7f38480d0590 con 0x7f384806bb30 2024-03-31T15:54:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384d400640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384806ec30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3840009dc0 tx=0x7f384001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797b97e0dc73cee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f3847e00640 1 -- 172.21.15.90:0/1910201792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38400b2070 con 0x7f38480cffb0 2024-03-31T15:54:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f3847e00640 1 -- 172.21.15.90:0/1910201792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38400a5950 con 0x7f38480cffb0 2024-03-31T15:54:59.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f3847e00640 1 -- 172.21.15.90:0/1910201792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38400a5d20 con 0x7f38480cffb0 2024-03-31T15:54:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 -- 172.21.15.90:0/1910201792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 msgr2=0x7f384806ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384806ec30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3840009dc0 tx=0x7f384001c890 comp rx=0 tx=0).stop 2024-03-31T15:54:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 -- 172.21.15.90:0/1910201792 shutdown_connections 2024-03-31T15:54:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384806ec30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 --2- 172.21.15.90:0/1910201792 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 -- 172.21.15.90:0/1910201792 >> 172.21.15.90:0/1910201792 conn(0x7f38480d0c60 msgr2=0x7f38480d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 -- 172.21.15.90:0/1910201792 shutdown_connections 2024-03-31T15:54:59.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.377+0000 7f384de00640 1 -- 172.21.15.90:0/1910201792 wait complete. 2024-03-31T15:54:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 Processor -- start 2024-03-31T15:54:59.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 -- start start 2024-03-31T15:54:59.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384806bb30 0x7f3848140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f384806fb00 con 0x7f38480cffb0 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f384806f980 con 0x7f384806bb30 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f384806fc80 con 0x7f38481456c0 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41822/0 (socket says 172.21.15.90:41822) 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 -- 172.21.15.90:0/2603722111 learned_addr learned my addr 172.21.15.90:0/2603722111 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384ca00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384806bb30 0x7f3848140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f384d400640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 -- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 msgr2=0x7f384815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 -- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384806bb30 msgr2=0x7f3848140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384806bb30 0x7f3848140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.381+0000 7f383fe00640 1 -- 172.21.15.90:0/2603722111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384813ede0 con 0x7f38480cffb0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f383fe00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f38300011f0 tx=0x7f383000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830018070 con 0x7f38480cffb0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38300074e0 con 0x7f38480cffb0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3830004a10 con 0x7f38480cffb0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384813f080 con 0x7f38480cffb0 2024-03-31T15:54:59.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384813f640 con 0x7f38480cffb0 2024-03-31T15:54:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3848068d00 con 0x7f38480cffb0 2024-03-31T15:54:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3830004bb0 con 0x7f38480cffb0 2024-03-31T15:54:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 0x7f382406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.385+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3830086120 con 0x7f38480cffb0 2024-03-31T15:54:59.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.389+0000 7f384ca00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 0x7f382406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.389+0000 7f384ca00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 0x7f382406a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f383800b930 tx=0x7f3838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.393+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3830056250 con 0x7f38480cffb0 2024-03-31T15:54:59.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.545+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f38480b0c90 con 0x7f38480cffb0 2024-03-31T15:54:59.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f3845600640 1 -- 172.21.15.90:0/2603722111 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3830055b50 con 0x7f38480cffb0 2024-03-31T15:54:59.540 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:54:59.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 msgr2=0x7f382406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 0x7f382406a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f383800b930 tx=0x7f3838008040 comp rx=0 tx=0).stop 2024-03-31T15:54:59.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 msgr2=0x7f384815af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f38300011f0 tx=0x7f383000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 shutdown_connections 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3824068380 0x7f382406a840 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38481456c0 0x7f384815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38480cffb0 0x7f384815af00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 --2- 172.21.15.90:0/2603722111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384806bb30 0x7f3848140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 >> 172.21.15.90:0/2603722111 conn(0x7f38480d0c60 msgr2=0x7f38480d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 shutdown_connections 2024-03-31T15:54:59.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.549+0000 7f384de00640 1 -- 172.21.15.90:0/2603722111 wait complete. 2024-03-31T15:54:59.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:54:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 Processor -- start 2024-03-31T15:54:59.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 -- start start 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7dc15d240 0x7fe7dc0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7dc0d0ea0 con 0x7fe7dc0ce910 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7dc0cd460 con 0x7fe7dc15d240 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7e2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7dc0cd740 con 0x7fe7dc163bd0 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7dbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7dc15d240 0x7fe7dc0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7db400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7dc0ce910 0x7fe7dc0cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:56928/0 (socket says 172.21.15.90:56928) 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7dc15d240 0x7fe7dc0ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:54502/0 (socket says 172.21.15.90:54502) 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7dbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:37074/0 (socket says 172.21.15.90:37074) 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7db400640 1 -- 172.21.15.90:0/3643595251 learned_addr learned my addr 172.21.15.90:0/3643595251 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 -- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 msgr2=0x7fe7dc165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 -- 172.21.15.90:0/3643595251 >> v1:172.21.15.102:6789/0 conn(0x7fe7dc0ce910 legacy=0x7fe7dc0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7d3e00640 1 -- 172.21.15.90:0/3643595251 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc0cda20 con 0x7fe7dc15d240 2024-03-31T15:54:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7dbe00640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:54:59.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.705+0000 7fe7daa00640 1 -- 172.21.15.90:0/3643595251 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2015120900 0 0) 0x7fe7dc0d0ea0 con 0x7fe7dc0ce910 2024-03-31T15:54:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7d3e00640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15d240 0x7fe7dc0ccb10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40097c0 tx=0x7fe7c401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29f765c43a976500 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7daa00640 1 -- 172.21.15.90:0/3643595251 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40ab600 con 0x7fe7dc15d240 2024-03-31T15:54:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7daa00640 1 -- 172.21.15.90:0/3643595251 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c40b0070 con 0x7fe7dc15d240 2024-03-31T15:54:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7daa00640 1 -- 172.21.15.90:0/3643595251 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40ab9c0 con 0x7fe7dc15d240 2024-03-31T15:54:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 -- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15d240 msgr2=0x7fe7dc0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15d240 0x7fe7dc0ccb10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40097c0 tx=0x7fe7c401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 -- 172.21.15.90:0/3643595251 shutdown_connections 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc163bd0 0x7fe7dc165fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 --2- 172.21.15.90:0/3643595251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15d240 0x7fe7dc0ccb10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 -- 172.21.15.90:0/3643595251 >> 172.21.15.90:0/3643595251 conn(0x7fe7dc08bfd0 msgr2=0x7fe7dc0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 -- 172.21.15.90:0/3643595251 shutdown_connections 2024-03-31T15:54:59.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.709+0000 7fe7e2600640 1 -- 172.21.15.90:0/3643595251 wait complete. 2024-03-31T15:54:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 Processor -- start 2024-03-31T15:54:59.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- start start 2024-03-31T15:54:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 0x7fe7dc0c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 0x7fe7dc1f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc168ea0 con 0x7fe7dc15d240 2024-03-31T15:54:59.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc168d20 con 0x7fe7dc163bd0 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7dc169020 con 0x7fe7dc0ce910 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41830/0 (socket says 172.21.15.90:41830) 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 -- 172.21.15.90:0/2675910933 learned_addr learned my addr 172.21.15.90:0/2675910933 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7db400640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 0x7fe7dc0c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7dbe00640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 0x7fe7dc1f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 -- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 msgr2=0x7fe7dc0c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 0x7fe7dc0c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 -- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 msgr2=0x7fe7dc1f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 0x7fe7dc1f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 -- 172.21.15.90:0/2675910933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc155e50 con 0x7fe7dc15d240 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7dbe00640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 0x7fe7dc1f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7db400640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 0x7fe7dc0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3e00640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40ad040 tx=0x7fe7c401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40bf410 con 0x7fe7dc15d240 2024-03-31T15:54:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c40bf5b0 con 0x7fe7dc15d240 2024-03-31T15:54:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7dc06cd40 con 0x7fe7dc15d240 2024-03-31T15:54:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c40bf9a0 con 0x7fe7dc15d240 2024-03-31T15:54:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.713+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7dc06d300 con 0x7fe7dc15d240 2024-03-31T15:54:59.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.717+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7dc068660 con 0x7fe7dc15d240 2024-03-31T15:54:59.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.717+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe7c40bd020 con 0x7fe7dc15d240 2024-03-31T15:54:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.717+0000 7fe7d3400640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 0x7fe7b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:54:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.717+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe7c4121330 con 0x7fe7dc15d240 2024-03-31T15:54:59.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.717+0000 7fe7db400640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 0x7fe7b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:54:59.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.721+0000 7fe7db400640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 0x7fe7b806a840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc007fd0 tx=0x7fe7cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:54:59.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.721+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe7c4016090 con 0x7fe7dc15d240 2024-03-31T15:54:59.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.905+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe7dc0ce6f0 con 0x7fe7dc15d240 2024-03-31T15:54:59.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.905+0000 7fe7d3400640 1 -- 172.21.15.90:0/2675910933 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fe7c40f1560 con 0x7fe7dc15d240 2024-03-31T15:54:59.899 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:54:59.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 msgr2=0x7fe7b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 0x7fe7b806a840 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc007fd0 tx=0x7fe7cc009040 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 msgr2=0x7fe7dc155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe7c40ad040 tx=0x7fe7c401ac20 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 shutdown_connections 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7b8068380 0x7fe7b806a840 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc163bd0 0x7fe7dc1f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc15d240 0x7fe7dc155910 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 --2- 172.21.15.90:0/2675910933 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0ce910 0x7fe7dc0c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 >> 172.21.15.90:0/2675910933 conn(0x7fe7dc08bfd0 msgr2=0x7fe7dc15b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 shutdown_connections 2024-03-31T15:54:59.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:54:59.909+0000 7fe7e2600640 1 -- 172.21.15.90:0/2675910933 wait complete. 2024-03-31T15:54:59.921 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:54:59.921 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:54:59.931 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:00.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.077+0000 7f013f200640 1 Processor -- start 2024-03-31T15:55:00.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.077+0000 7f013f200640 1 -- start start 2024-03-31T15:55:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f013815c120 0x7f013815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 0x7f01380b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013f200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01380d2b10 con 0x7f01380b2c20 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0138160bf0 con 0x7f013815c120 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0138160350 con 0x7f01380d2680 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f013815c120 0x7f013815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013d200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f01380b2c20 0x7f01380cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50234/0 (socket says 172.21.15.90:50234) 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f013815c120 0x7f013815fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41832/0 (socket says 172.21.15.90:41832) 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f013d200640 1 -- 172.21.15.90:0/350649971 learned_addr learned my addr 172.21.15.90:0/350649971 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137400640 1 --2- 172.21.15.90:0/350649971 >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 0x7f01380b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 -- 172.21.15.90:0/350649971 >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 msgr2=0x7f01380b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 --2- 172.21.15.90:0/350649971 >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 0x7f01380b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 -- 172.21.15.90:0/350649971 >> v1:172.21.15.90:6789/0 conn(0x7f01380b2c20 legacy=0x7f01380cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137e00640 1 -- 172.21.15.90:0/350649971 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0138160630 con 0x7f013815c120 2024-03-31T15:55:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.081+0000 7f0137400640 1 --2- 172.21.15.90:0/350649971 >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 0x7f01380b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f0137e00640 1 --2- 172.21.15.90:0/350649971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f013815c120 0x7f013815fd00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f013806ed50 tx=0x7f012801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc680305efa0644e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f0136a00640 1 -- 172.21.15.90:0/350649971 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01280b1070 con 0x7f013815c120 2024-03-31T15:55:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f0136a00640 1 -- 172.21.15.90:0/350649971 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01280ac620 con 0x7f013815c120 2024-03-31T15:55:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f0136a00640 1 -- 172.21.15.90:0/350649971 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01280ac9f0 con 0x7f013815c120 2024-03-31T15:55:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- 172.21.15.90:0/350649971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f013815c120 msgr2=0x7f013815fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- 172.21.15.90:0/350649971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f013815c120 0x7f013815fd00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f013806ed50 tx=0x7f012801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- 172.21.15.90:0/350649971 shutdown_connections 2024-03-31T15:55:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- 172.21.15.90:0/350649971 >> v2:172.21.15.102:3300/0 conn(0x7f01380d2680 0x7f01380b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- 172.21.15.90:0/350649971 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f013815c120 0x7f013815fd00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- 172.21.15.90:0/350649971 >> 172.21.15.90:0/350649971 conn(0x7f013809aa40 msgr2=0x7f013809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- 172.21.15.90:0/350649971 shutdown_connections 2024-03-31T15:55:00.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- 172.21.15.90:0/350649971 wait complete. 2024-03-31T15:55:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 Processor -- start 2024-03-31T15:55:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- start start 2024-03-31T15:55:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 0x7f0138152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 0x7f0138153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f01381691c0 con 0x7f01380d2680 2024-03-31T15:55:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0138169040 con 0x7f01380b2c20 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.085+0000 7f013f200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0138169340 con 0x7f013815c120 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41838/0 (socket says 172.21.15.90:41838) 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 -- 172.21.15.90:0/1153429459 learned_addr learned my addr 172.21.15.90:0/1153429459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137e00640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 0x7f0138152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f013d200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 0x7f0138153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 -- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 msgr2=0x7f0138153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 0x7f0138153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 -- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 msgr2=0x7f0138152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 0x7f0138152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 -- 172.21.15.90:0/1153429459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01380aef70 con 0x7f01380d2680 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137e00640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 0x7f0138152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f013d200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 0x7f0138153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f0137400640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f012400d830 tx=0x7f012400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124018020 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01240096c0 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124009ab0 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01380af250 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01380af810 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01380686d0 con 0x7f01380d2680 2024-03-31T15:55:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f012401b020 con 0x7f01380d2680 2024-03-31T15:55:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.089+0000 7f012fe00640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 0x7f011806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.093+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f01240875f0 con 0x7f01380d2680 2024-03-31T15:55:00.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.093+0000 7f0137e00640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 0x7f011806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.093+0000 7f0137e00640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 0x7f011806a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01280ae000 tx=0x7f0128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.097+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0124022020 con 0x7f01380d2680 2024-03-31T15:55:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.253+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f013815f810 con 0x7f01380d2680 2024-03-31T15:55:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.253+0000 7f012fe00640 1 -- 172.21.15.90:0/1153429459 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f0124057790 con 0x7f01380d2680 2024-03-31T15:55:00.248 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:00.248 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 msgr2=0x7f011806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 0x7f011806a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01280ae000 tx=0x7f0128008040 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 msgr2=0x7f013815c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f012400d830 tx=0x7f012400dcf0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 shutdown_connections 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0118068380 0x7f011806a840 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f013815c120 0x7f0138153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01380d2680 0x7f013815c7e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 --2- 172.21.15.90:0/1153429459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380b2c20 0x7f0138152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 >> 172.21.15.90:0/1153429459 conn(0x7f013809aa40 msgr2=0x7f0138158d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 shutdown_connections 2024-03-31T15:55:00.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.257+0000 7f013f200640 1 -- 172.21.15.90:0/1153429459 wait complete. 2024-03-31T15:55:00.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:55:00.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.421+0000 7fdf9f000640 1 Processor -- start 2024-03-31T15:55:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.421+0000 7fdf9f000640 1 -- start start 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf9f000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf9815e4e0 0x7fdf9806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf9f000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf9806ce50 0x7fdf9806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf9f000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9815e8c0 con 0x7fdf980ce7a0 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf9f000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdf980d0ff0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf9f000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdf980d12d0 con 0x7fdf9806ce50 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf9815e4e0 0x7fdf9806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdf9815e4e0 0x7fdf9806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41842/0 (socket says 172.21.15.90:41842) 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 -- 172.21.15.90:0/1031047403 learned_addr learned my addr 172.21.15.90:0/1031047403 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf8fe00640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf9806ce50 0x7fdf9806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 -- 172.21.15.90:0/1031047403 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf9806ce50 msgr2=0x7fdf9806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:00.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf9806ce50 0x7fdf9806d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 -- 172.21.15.90:0/1031047403 >> v1:172.21.15.102:6789/0 conn(0x7fdf980ce7a0 legacy=0x7fdf980cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 -- 172.21.15.90:0/1031047403 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf980cc710 con 0x7fdf9815e4e0 2024-03-31T15:55:00.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.425+0000 7fdf97e00640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9806c910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009870 tx=0x7fdf8801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62d7dffd0a3e898 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf97400640 1 -- 172.21.15.90:0/1031047403 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880b0070 con 0x7fdf9815e4e0 2024-03-31T15:55:00.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf97400640 1 -- 172.21.15.90:0/1031047403 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf880ab600 con 0x7fdf9815e4e0 2024-03-31T15:55:00.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf97400640 1 -- 172.21.15.90:0/1031047403 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880ab9d0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- 172.21.15.90:0/1031047403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 msgr2=0x7fdf9806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9806c910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009870 tx=0x7fdf8801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- 172.21.15.90:0/1031047403 shutdown_connections 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf9806ce50 0x7fdf9806d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1031047403 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9806c910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- 172.21.15.90:0/1031047403 >> 172.21.15.90:0/1031047403 conn(0x7fdf980d1f20 msgr2=0x7fdf980d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- 172.21.15.90:0/1031047403 shutdown_connections 2024-03-31T15:55:00.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- 172.21.15.90:0/1031047403 wait complete. 2024-03-31T15:55:00.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 Processor -- start 2024-03-31T15:55:00.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- start start 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9806ce50 0x7fdf98155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf980ce7a0 0x7fdf98149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdf980d0d90 con 0x7fdf9815e4e0 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdf980d0c10 con 0x7fdf9806ce50 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.429+0000 7fdf9f000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9816b840 con 0x7fdf980ce7a0 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41844/0 (socket says 172.21.15.90:41844) 2024-03-31T15:55:00.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 -- 172.21.15.90:0/1783503911 learned_addr learned my addr 172.21.15.90:0/1783503911 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:00.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf8fe00640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf980ce7a0 0x7fdf98149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf97e00640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9806ce50 0x7fdf98155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 -- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf980ce7a0 msgr2=0x7fdf98149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf980ce7a0 0x7fdf98149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 -- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9806ce50 msgr2=0x7fdf98155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9806ce50 0x7fdf98155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 -- 172.21.15.90:0/1783503911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf980c6810 con 0x7fdf9815e4e0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9d000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c7a0 tx=0x7fdf9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf90003b80 con 0x7fdf9815e4e0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf90003d20 con 0x7fdf9815e4e0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9000aa90 con 0x7fdf9815e4e0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf981564b0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf98155ef0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf980685f0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.433+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdf9000ac30 con 0x7fdf9815e4e0 2024-03-31T15:55:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.437+0000 7fdf94c00640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 0x7fdf7406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.437+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdf90086b70 con 0x7fdf9815e4e0 2024-03-31T15:55:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.437+0000 7fdf97e00640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 0x7fdf7406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.437+0000 7fdf97e00640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 0x7fdf7406a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf880ad000 tx=0x7fdf880a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.445+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdf900566d0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf98157840 con 0x7fdf9815e4e0 2024-03-31T15:55:00.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf94c00640 1 -- 172.21.15.90:0/1783503911 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fdf9005a9d0 con 0x7fdf9815e4e0 2024-03-31T15:55:00.594 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 msgr2=0x7fdf7406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 0x7fdf7406a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf880ad000 tx=0x7fdf880a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 msgr2=0x7fdf9815bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.601+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000c7a0 tx=0x7fdf9000cc60 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 shutdown_connections 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdf74068380 0x7fdf7406a840 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf9815e4e0 0x7fdf9815bff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdf980ce7a0 0x7fdf98149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 --2- 172.21.15.90:0/1783503911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9806ce50 0x7fdf98155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 >> 172.21.15.90:0/1783503911 conn(0x7fdf980d1f20 msgr2=0x7fdf980c6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 shutdown_connections 2024-03-31T15:55:00.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.605+0000 7fdf9f000640 1 -- 172.21.15.90:0/1783503911 wait complete. 2024-03-31T15:55:00.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.761+0000 7fb6b3200640 1 Processor -- start 2024-03-31T15:55:00.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.761+0000 7fb6b3200640 1 -- start start 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b3200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6ac15e4e0 0x7fb6ac06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b3200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6ac0cdf20 0x7fb6ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b3200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6ac15e8c0 con 0x7fb6ac06ce50 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b3200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ac0d0ff0 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b3200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ac0d12d0 con 0x7fb6ac0cdf20 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6ac15e4e0 0x7fb6ac06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb6ac15e4e0 0x7fb6ac06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41848/0 (socket says 172.21.15.90:41848) 2024-03-31T15:55:00.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 -- 172.21.15.90:0/1806101273 learned_addr learned my addr 172.21.15.90:0/1806101273 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6ab400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb6ac06ce50 0x7fb6ac0ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50250/0 (socket says 172.21.15.90:50250) 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6b1200640 1 --2- 172.21.15.90:0/1806101273 >> v2:172.21.15.102:3300/0 conn(0x7fb6ac0cdf20 0x7fb6ac06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 -- 172.21.15.90:0/1806101273 >> v2:172.21.15.102:3300/0 conn(0x7fb6ac0cdf20 msgr2=0x7fb6ac06d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 --2- 172.21.15.90:0/1806101273 >> v2:172.21.15.102:3300/0 conn(0x7fb6ac0cdf20 0x7fb6ac06d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 -- 172.21.15.90:0/1806101273 >> v1:172.21.15.90:6789/0 conn(0x7fb6ac06ce50 legacy=0x7fb6ac0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6abe00640 1 -- 172.21.15.90:0/1806101273 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac0cc710 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.765+0000 7fb6aaa00640 1 -- 172.21.15.90:0/1806101273 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225383774 0 0) 0x7fb6ac15e8c0 con 0x7fb6ac06ce50 2024-03-31T15:55:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6abe00640 1 --2- 172.21.15.90:0/1806101273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac06c910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009870 tx=0x7fb69c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65c1ee05a72a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6aaa00640 1 -- 172.21.15.90:0/1806101273 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0b1070 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6aaa00640 1 -- 172.21.15.90:0/1806101273 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb69c0ac600 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6aaa00640 1 -- 172.21.15.90:0/1806101273 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0ac9d0 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- 172.21.15.90:0/1806101273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 msgr2=0x7fb6ac06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 --2- 172.21.15.90:0/1806101273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac06c910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009870 tx=0x7fb69c01c490 comp rx=0 tx=0).stop 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- 172.21.15.90:0/1806101273 shutdown_connections 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 --2- 172.21.15.90:0/1806101273 >> v2:172.21.15.102:3300/0 conn(0x7fb6ac0cdf20 0x7fb6ac06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 --2- 172.21.15.90:0/1806101273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac06c910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- 172.21.15.90:0/1806101273 >> 172.21.15.90:0/1806101273 conn(0x7fb6ac0d1f20 msgr2=0x7fb6ac0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- 172.21.15.90:0/1806101273 shutdown_connections 2024-03-31T15:55:00.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- 172.21.15.90:0/1806101273 wait complete. 2024-03-31T15:55:00.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 Processor -- start 2024-03-31T15:55:00.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.769+0000 7fb6b3200640 1 -- start start 2024-03-31T15:55:00.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 0x7fb6ac0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 0x7fb6ac15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac16b1b0 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac16b030 con 0x7fb6ac0cdf20 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac16b330 con 0x7fb6ac06ce50 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41854/0 (socket says 172.21.15.90:41854) 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 -- 172.21.15.90:0/2078645715 learned_addr learned my addr 172.21.15.90:0/2078645715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6ab400640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 0x7fb6ac15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6abe00640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 0x7fb6ac0c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 -- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 msgr2=0x7fb6ac0c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 0x7fb6ac0c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 -- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 msgr2=0x7fb6ac15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 0x7fb6ac15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 -- 172.21.15.90:0/2078645715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac0cb330 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6abe00640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 0x7fb6ac0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6ab400640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 0x7fb6ac15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b1200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400efc0 tx=0x7fb6a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4018020 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a4014480 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4014870 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac0cb610 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac15b260 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.773+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ac15b460 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.777+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb6a401b020 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.777+0000 7fb6a3e00640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 0x7fb68c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.777+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb6a4087c20 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.777+0000 7fb6abe00640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 0x7fb68c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:00.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.777+0000 7fb6abe00640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 0x7fb68c06a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0ae000 tx=0x7fb69c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:00.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.781+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb6ac15b460 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.937+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb6ac0685f0 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.937+0000 7fb6a3e00640 1 -- 172.21.15.90:0/2078645715 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fb6a4022030 con 0x7fb6ac15e4e0 2024-03-31T15:55:00.931 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 msgr2=0x7fb68c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 0x7fb68c06a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0ae000 tx=0x7fb69c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 msgr2=0x7fb6ac15bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400efc0 tx=0x7fb6a400c490 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 shutdown_connections 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb68c068380 0x7fb68c06a840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6ac15e4e0 0x7fb6ac15bcc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6ac0cdf20 0x7fb6ac15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 --2- 172.21.15.90:0/2078645715 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb6ac06ce50 0x7fb6ac0c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 >> 172.21.15.90:0/2078645715 conn(0x7fb6ac0d1f20 msgr2=0x7fb6ac156b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:00.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 shutdown_connections 2024-03-31T15:55:00.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:00.941+0000 7fb6b3200640 1 -- 172.21.15.90:0/2078645715 wait complete. 2024-03-31T15:55:00.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:01.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 Processor -- start 2024-03-31T15:55:01.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 -- start start 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 0x7f3e3015f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e3015fa10 con 0x7f3e30072370 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e3015b3b0 con 0x7f3e300b3d50 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e38800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3e300aea10 con 0x7f3e3015afd0 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e36800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3e30072370 0x7f3e300afab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50254/0 (socket says 172.21.15.90:50254) 2024-03-31T15:55:01.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e36800640 1 -- 172.21.15.90:0/4265295746 learned_addr learned my addr 172.21.15.90:0/4265295746 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e35e00640 1 --2- 172.21.15.90:0/4265295746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 0x7f3e3015f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672311739 0 0) 0x7f3e3015fa10 con 0x7f3e30072370 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e18003530 con 0x7f3e30072370 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012550747 0 0) 0x7f3e3015b3b0 con 0x7f3e300b3d50 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e3015fa10 con 0x7f3e300b3d50 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1662937895 0 0) 0x7f3e18003530 con 0x7f3e30072370 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e35e00640 1 -- 172.21.15.90:0/4265295746 >> v1:172.21.15.102:6789/0 conn(0x7f3e300b3d50 legacy=0x7f3e3006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.125+0000 7f3e35e00640 1 -- 172.21.15.90:0/4265295746 >> v1:172.21.15.90:6789/0 conn(0x7f3e30072370 legacy=0x7f3e300afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e35e00640 1 -- 172.21.15.90:0/4265295746 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e300aecf0 con 0x7f3e3015afd0 2024-03-31T15:55:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e35e00640 1 --2- 172.21.15.90:0/4265295746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 0x7f3e3015f4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097f0 tx=0x7f3e2401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1086090ddd611aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e240b0070 con 0x7f3e3015afd0 2024-03-31T15:55:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e240ab400 con 0x7f3e3015afd0 2024-03-31T15:55:01.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e34a00640 1 -- 172.21.15.90:0/4265295746 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c001a00 con 0x7f3e3015afd0 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 -- 172.21.15.90:0/4265295746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 msgr2=0x7f3e3015f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 --2- 172.21.15.90:0/4265295746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 0x7f3e3015f4d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e240097f0 tx=0x7f3e2401c400 comp rx=0 tx=0).stop 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 -- 172.21.15.90:0/4265295746 shutdown_connections 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 --2- 172.21.15.90:0/4265295746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e3015afd0 0x7f3e3015f4d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 -- 172.21.15.90:0/4265295746 >> 172.21.15.90:0/4265295746 conn(0x7f3e300b29c0 msgr2=0x7f3e300b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 -- 172.21.15.90:0/4265295746 shutdown_connections 2024-03-31T15:55:01.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.129+0000 7f3e38800640 1 -- 172.21.15.90:0/4265295746 wait complete. 2024-03-31T15:55:01.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 Processor -- start 2024-03-31T15:55:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 -- start start 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e300b3d50 0x7f3e3015c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3015afd0 0x7f3e301577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30072c10 con 0x7f3e30072370 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3e300b05d0 con 0x7f3e3015afd0 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e38800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3e30169680 con 0x7f3e300b3d50 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e35e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e35e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41860/0 (socket says 172.21.15.90:41860) 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e35e00640 1 -- 172.21.15.90:0/1756688076 learned_addr learned my addr 172.21.15.90:0/1756688076 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e36800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3015afd0 0x7f3e301577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.133+0000 7f3e35400640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e300b3d50 0x7f3e3015c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 -- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e300b3d50 msgr2=0x7f3e3015c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e300b3d50 0x7f3e3015c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 -- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3015afd0 msgr2=0x7f3e301577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3015afd0 0x7f3e301577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 -- 172.21.15.90:0/1756688076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3014fef0 con 0x7f3e30072370 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35e00640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3e240ae040 tx=0x7f3e2401ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e240b0050 con 0x7f3e30072370 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e240b1040 con 0x7f3e30072370 2024-03-31T15:55:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004280 con 0x7f3e30072370 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3015d3d0 con 0x7f3e30072370 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3015d990 con 0x7f3e30072370 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e300685f0 con 0x7f3e30072370 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3e240be020 con 0x7f3e30072370 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 0x7f3e1406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3e24120cb0 con 0x7f3e30072370 2024-03-31T15:55:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.137+0000 7f3e35400640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 0x7f3e1406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.141+0000 7f3e35400640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 0x7f3e1406a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006f2b0 tx=0x7f3e20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.141+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3e24015ff0 con 0x7f3e30072370 2024-03-31T15:55:01.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.301+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3e301533a0 con 0x7f3e30072370 2024-03-31T15:55:01.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.301+0000 7f3e2a000640 1 -- 172.21.15.90:0/1756688076 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f3e240f0e60 con 0x7f3e30072370 2024-03-31T15:55:01.296 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 msgr2=0x7f3e1406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 0x7f3e1406a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006f2b0 tx=0x7f3e20009040 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 msgr2=0x7f3e3014f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3e240ae040 tx=0x7f3e2401ac40 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 shutdown_connections 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e140683b0 0x7f3e1406a870 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3015afd0 0x7f3e301577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3e300b3d50 0x7f3e3015c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 --2- 172.21.15.90:0/1756688076 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30072370 0x7f3e3014f9b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 >> 172.21.15.90:0/1756688076 conn(0x7f3e300b29c0 msgr2=0x7f3e30152650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 shutdown_connections 2024-03-31T15:55:01.298 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.305+0000 7f3e38800640 1 -- 172.21.15.90:0/1756688076 wait complete. 2024-03-31T15:55:01.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:55:01.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.469+0000 7fd83d400640 1 Processor -- start 2024-03-31T15:55:01.464 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.469+0000 7fd83d400640 1 -- start start 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd83d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8380d2e60 0x7fd8380b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd83d400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8380b71d0 con 0x7fd83815e6a0 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd83d400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8380b74b0 con 0x7fd8380cfe50 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd83d400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd8380b7790 con 0x7fd8380d2e60 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd836800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd83815e6a0 0x7fd8380cf740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50262/0 (socket says 172.21.15.90:50262) 2024-03-31T15:55:01.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd836800640 1 -- 172.21.15.90:0/1668074603 learned_addr learned my addr 172.21.15.90:0/1668074603 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:01.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd837200640 1 --2- 172.21.15.90:0/1668074603 >> v2:172.21.15.102:3300/0 conn(0x7fd8380d2e60 0x7fd8380b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948712742 0 0) 0x7fd8380b71d0 con 0x7fd83815e6a0 2024-03-31T15:55:01.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd81c003530 con 0x7fd83815e6a0 2024-03-31T15:55:01.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178256664 0 0) 0x7fd8380b74b0 con 0x7fd8380cfe50 2024-03-31T15:55:01.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8380b71d0 con 0x7fd8380cfe50 2024-03-31T15:55:01.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd837200640 1 -- 172.21.15.90:0/1668074603 >> v1:172.21.15.102:6789/0 conn(0x7fd8380cfe50 legacy=0x7fd8380d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd837200640 1 -- 172.21.15.90:0/1668074603 >> v1:172.21.15.90:6789/0 conn(0x7fd83815e6a0 legacy=0x7fd8380cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.473+0000 7fd837200640 1 -- 172.21.15.90:0/1668074603 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8380b7a70 con 0x7fd8380d2e60 2024-03-31T15:55:01.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd837200640 1 --2- 172.21.15.90:0/1668074603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380b4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd830009d90 tx=0x7fd83001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=473c40196c38e38a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8300b4070 con 0x7fd8380d2e60 2024-03-31T15:55:01.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8300af600 con 0x7fd8380d2e60 2024-03-31T15:55:01.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd835400640 1 -- 172.21.15.90:0/1668074603 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8300af9d0 con 0x7fd8380d2e60 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- 172.21.15.90:0/1668074603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 msgr2=0x7fd8380b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 --2- 172.21.15.90:0/1668074603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380b4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd830009d90 tx=0x7fd83001f150 comp rx=0 tx=0).stop 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- 172.21.15.90:0/1668074603 shutdown_connections 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 --2- 172.21.15.90:0/1668074603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380b4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- 172.21.15.90:0/1668074603 >> 172.21.15.90:0/1668074603 conn(0x7fd83809d4b0 msgr2=0x7fd8380d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- 172.21.15.90:0/1668074603 shutdown_connections 2024-03-31T15:55:01.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- 172.21.15.90:0/1668074603 wait complete. 2024-03-31T15:55:01.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 Processor -- start 2024-03-31T15:55:01.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.477+0000 7fd83d400640 1 -- start start 2024-03-31T15:55:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 0x7fd8381f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd8380b9400 con 0x7fd83815e6a0 2024-03-31T15:55:01.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd8380b9280 con 0x7fd8380d2e60 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd8380b9580 con 0x7fd8380cfe50 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41870/0 (socket says 172.21.15.90:41870) 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 -- 172.21.15.90:0/3341333988 learned_addr learned my addr 172.21.15.90:0/3341333988 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd835e00640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd836800640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 0x7fd8381f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 -- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 msgr2=0x7fd8381f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 0x7fd8381f4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 -- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 msgr2=0x7fd8380a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 -- 172.21.15.90:0/3341333988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8381f48d0 con 0x7fd83815e6a0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd836800640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 0x7fd8381f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd837200640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd8300b2040 tx=0x7fd83001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8300b4040 con 0x7fd83815e6a0 2024-03-31T15:55:01.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8300c1070 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd830009450 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8381f4bb0 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8381f4590 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8380685f0 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.481+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8300ba6b0 con 0x7fd83815e6a0 2024-03-31T15:55:01.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.485+0000 7fd82ea00640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 0x7fd81806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.485+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd830125620 con 0x7fd83815e6a0 2024-03-31T15:55:01.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.485+0000 7fd836800640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 0x7fd81806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.485+0000 7fd836800640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 0x7fd81806a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd828000f70 tx=0x7fd828009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.493+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd830018cb0 con 0x7fd83815e6a0 2024-03-31T15:55:01.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd8380d2530 con 0x7fd83815e6a0 2024-03-31T15:55:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd82ea00640 1 -- 172.21.15.90:0/3341333988 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fd8300f57d0 con 0x7fd83815e6a0 2024-03-31T15:55:01.645 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 msgr2=0x7fd81806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 0x7fd81806a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd828000f70 tx=0x7fd828009040 comp rx=0 tx=0).stop 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 msgr2=0x7fd83815cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd8300b2040 tx=0x7fd83001bef0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 shutdown_connections 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd818068380 0x7fd81806a840 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd83815e6a0 0x7fd83815cfc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8380d2e60 0x7fd8380a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 --2- 172.21.15.90:0/3341333988 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd8380cfe50 0x7fd8381f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 >> 172.21.15.90:0/3341333988 conn(0x7fd83809d4b0 msgr2=0x7fd8380d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 shutdown_connections 2024-03-31T15:55:01.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.653+0000 7fd83d400640 1 -- 172.21.15.90:0/3341333988 wait complete. 2024-03-31T15:55:01.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:55:01.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 Processor -- start 2024-03-31T15:55:01.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 -- start start 2024-03-31T15:55:01.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04240ccd80 0x7f042406e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04240cd1e0 con 0x7f042406e950 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04240cbd20 con 0x7f042415c1d0 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f042a200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f042406ed80 con 0x7f04240ccd80 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f041be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f04240ccd80 0x7f042406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f0423e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f042406e950 0x7f04240b2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50268/0 (socket says 172.21.15.90:50268) 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.809+0000 7f0423e00640 1 -- 172.21.15.90:0/1436551932 learned_addr learned my addr 172.21.15.90:0/1436551932 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1228605834 0 0) 0x7f041c0014a0 con 0x7f042406e950 2024-03-31T15:55:01.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0408003530 con 0x7f042406e950 2024-03-31T15:55:01.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2870978040 0 0) 0x7f04240cbd20 con 0x7f042415c1d0 2024-03-31T15:55:01.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f041be00640 1 -- 172.21.15.90:0/1436551932 >> v1:172.21.15.102:6789/0 conn(0x7f042415c1d0 legacy=0x7f0424072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f041be00640 1 -- 172.21.15.90:0/1436551932 >> v1:172.21.15.90:6789/0 conn(0x7f042406e950 legacy=0x7f04240b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f041be00640 1 -- 172.21.15.90:0/1436551932 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04240cbfc0 con 0x7f04240ccd80 2024-03-31T15:55:01.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f041be00640 1 --2- 172.21.15.90:0/1436551932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f042406e410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f040c0097c0 tx=0x7f040c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbbd173ea7b53d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0b0070 con 0x7f04240ccd80 2024-03-31T15:55:01.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f040c0ab600 con 0x7f04240ccd80 2024-03-31T15:55:01.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f0422a00640 1 -- 172.21.15.90:0/1436551932 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0ab9d0 con 0x7f04240ccd80 2024-03-31T15:55:01.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f042a200640 1 -- 172.21.15.90:0/1436551932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 msgr2=0x7f042406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.813+0000 7f042a200640 1 --2- 172.21.15.90:0/1436551932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f042406e410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f040c0097c0 tx=0x7f040c01c410 comp rx=0 tx=0).stop 2024-03-31T15:55:01.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- 172.21.15.90:0/1436551932 shutdown_connections 2024-03-31T15:55:01.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 --2- 172.21.15.90:0/1436551932 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f042406e410 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- 172.21.15.90:0/1436551932 >> 172.21.15.90:0/1436551932 conn(0x7f04240b1a50 msgr2=0x7f04240b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- 172.21.15.90:0/1436551932 shutdown_connections 2024-03-31T15:55:01.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- 172.21.15.90:0/1436551932 wait complete. 2024-03-31T15:55:01.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 Processor -- start 2024-03-31T15:55:01.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- start start 2024-03-31T15:55:01.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042406e950 0x7f04240c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f042415c1d0 0x7f04240ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0424073440 con 0x7f04240ccd80 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0424071450 con 0x7f042406e950 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f042a200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f04240b36a0 con 0x7f042415c1d0 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41884/0 (socket says 172.21.15.90:41884) 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 -- 172.21.15.90:0/2350802324 learned_addr learned my addr 172.21.15.90:0/2350802324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f0423400640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042406e950 0x7f04240c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f0423e00640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f042415c1d0 0x7f04240ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 -- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f042415c1d0 msgr2=0x7f04240ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f042415c1d0 0x7f04240ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 -- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042406e950 msgr2=0x7f04240c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042406e950 0x7f04240c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 -- 172.21.15.90:0/2350802324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04240ce230 con 0x7f04240ccd80 2024-03-31T15:55:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.817+0000 7f041be00640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f040c0097c0 tx=0x7f040c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0b8070 con 0x7f04240ccd80 2024-03-31T15:55:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f040c004020 con 0x7f04240ccd80 2024-03-31T15:55:01.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0b0040 con 0x7f04240ccd80 2024-03-31T15:55:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04240ce510 con 0x7f04240ccd80 2024-03-31T15:55:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04240cead0 con 0x7f04240ccd80 2024-03-31T15:55:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0424068660 con 0x7f04240ccd80 2024-03-31T15:55:01.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f040c003620 con 0x7f04240ccd80 2024-03-31T15:55:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f040c1207a0 con 0x7f04240ccd80 2024-03-31T15:55:01.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.821+0000 7f0423400640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:01.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.825+0000 7f0423400640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 0x7f040006a870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0414000f70 tx=0x7f0414009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:01.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.829+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f040c016090 con 0x7f04240ccd80 2024-03-31T15:55:01.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:01.997+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f042415ae90 con 0x7f04240ccd80 2024-03-31T15:55:01.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f041b400640 1 -- 172.21.15.90:0/2350802324 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f040c0f03a0 con 0x7f04240ccd80 2024-03-31T15:55:01.993 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 msgr2=0x7f040006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 0x7f040006a870 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0414000f70 tx=0x7f0414009040 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 msgr2=0x7f0424145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f040c0097c0 tx=0x7f040c0191b0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 shutdown_connections 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f04000683b0 0x7f040006a870 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f042415c1d0 0x7f04240ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04240ccd80 0x7f0424145f10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 --2- 172.21.15.90:0/2350802324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f042406e950 0x7f04240c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:01.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 >> 172.21.15.90:0/2350802324 conn(0x7f04240b1a50 msgr2=0x7f04240b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:01.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 shutdown_connections 2024-03-31T15:55:01.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.001+0000 7f042a200640 1 -- 172.21.15.90:0/2350802324 wait complete. 2024-03-31T15:55:02.022 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com... 2024-03-31T15:55:02.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:02.022 DEBUG:teuthology.orchestra.run.smithi102:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-31T15:55:02.044 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:02.044 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:55:02.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:02.044 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:55:02.094 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:55:02.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:02.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 Processor -- start 2024-03-31T15:55:02.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 -- start start 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265406e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2654073590 0x7f265415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f265406ce30 con 0x7f265406ee20 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f265406f2c0 con 0x7f2654073590 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f265406f5a0 con 0x7f265415af10 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2654073590 0x7f265415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2654073590 0x7f265415b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41896/0 (socket says 172.21.15.90:41896) 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 -- 172.21.15.90:0/1620238906 learned_addr learned my addr 172.21.15.90:0/1620238906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:02.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2653400640 1 --1- 172.21.15.90:0/1620238906 >> v1:172.21.15.90:6789/0 conn(0x7f265406ee20 0x7f265415b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50272/0 (socket says 172.21.15.90:50272) 2024-03-31T15:55:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652a00640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265406e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 -- 172.21.15.90:0/1620238906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 msgr2=0x7f265406e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265406e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 -- 172.21.15.90:0/1620238906 >> v1:172.21.15.90:6789/0 conn(0x7f265406ee20 legacy=0x7f265415b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.277+0000 7f2652000640 1 -- 172.21.15.90:0/1620238906 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265406f880 con 0x7f2654073590 2024-03-31T15:55:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2652000640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2654073590 0x7f265415b2f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2640009870 tx=0x7f264001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec858182afec448 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2651600640 1 -- 172.21.15.90:0/1620238906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26400b0070 con 0x7f2654073590 2024-03-31T15:55:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2651600640 1 -- 172.21.15.90:0/1620238906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26400ab600 con 0x7f2654073590 2024-03-31T15:55:02.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2651600640 1 -- 172.21.15.90:0/1620238906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26400ab9d0 con 0x7f2654073590 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- 172.21.15.90:0/1620238906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2654073590 msgr2=0x7f265415b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2654073590 0x7f265415b2f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2640009870 tx=0x7f264001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- 172.21.15.90:0/1620238906 shutdown_connections 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2654073590 0x7f265415b2f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 --2- 172.21.15.90:0/1620238906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265406e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- 172.21.15.90:0/1620238906 >> 172.21.15.90:0/1620238906 conn(0x7f26540cff30 msgr2=0x7f26540d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- 172.21.15.90:0/1620238906 shutdown_connections 2024-03-31T15:55:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- 172.21.15.90:0/1620238906 wait complete. 2024-03-31T15:55:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 Processor -- start 2024-03-31T15:55:02.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.281+0000 7f2653e00640 1 -- start start 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f265414cc10 0x7f26540b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f26540721b0 con 0x7f265406ee20 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2654072030 con 0x7f265414cc10 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2654072330 con 0x7f265415af10 2024-03-31T15:55:02.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41904/0 (socket says 172.21.15.90:41904) 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 -- 172.21.15.90:0/2292280365 learned_addr learned my addr 172.21.15.90:0/2292280365 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652000640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653400640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f265414cc10 0x7f26540b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 -- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 msgr2=0x7f265414c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265414c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 -- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f265414cc10 msgr2=0x7f26540b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f265414cc10 0x7f26540b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 -- 172.21.15.90:0/2292280365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265415e140 con 0x7f265406ee20 2024-03-31T15:55:02.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2652a00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f264400d830 tx=0x7f264400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264400b6c0 con 0x7f265406ee20 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2644019070 con 0x7f265406ee20 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264400ba60 con 0x7f265406ee20 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265415e420 con 0x7f265406ee20 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26541489c0 con 0x7f265406ee20 2024-03-31T15:55:02.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2654068d00 con 0x7f265406ee20 2024-03-31T15:55:02.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.285+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f264401e460 con 0x7f265406ee20 2024-03-31T15:55:02.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.289+0000 7f264aa00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 0x7f263406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.289+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2644087520 con 0x7f265406ee20 2024-03-31T15:55:02.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.289+0000 7f2652000640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 0x7f263406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.289+0000 7f2652000640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 0x7f263406a870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26400ad000 tx=0x7f2640008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.293+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2644057640 con 0x7f265406ee20 2024-03-31T15:55:02.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.485+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26540d0b20 con 0x7f265406ee20 2024-03-31T15:55:02.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.485+0000 7f264aa00640 1 -- 172.21.15.90:0/2292280365 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f2644056f40 con 0x7f265406ee20 2024-03-31T15:55:02.479 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:02.479 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:55:02.480 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:55:02.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 msgr2=0x7f263406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 0x7f263406a870 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f26400ad000 tx=0x7f2640008040 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 msgr2=0x7f2654143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f264400d830 tx=0x7f264400dcf0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 shutdown_connections 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f26340683b0 0x7f263406a870 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f265414cc10 0x7f26540b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f265415af10 0x7f265414c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 --2- 172.21.15.90:0/2292280365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f265406ee20 0x7f2654143fd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 >> 172.21.15.90:0/2292280365 conn(0x7f26540cff30 msgr2=0x7f26540ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 shutdown_connections 2024-03-31T15:55:02.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.489+0000 7f2653e00640 1 -- 172.21.15.90:0/2292280365 wait complete. 2024-03-31T15:55:02.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:02.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 Processor -- start 2024-03-31T15:55:02.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 -- start start 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15141449a0 0x7f151415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15140b3d90 0x7f15140b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15140b47e0 con 0x7f151406b820 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f15140b4ac0 con 0x7f15140b3d90 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151c000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f151415b7d0 con 0x7f15141449a0 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151a000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f151406b820 0x7f151415ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50280/0 (socket says 172.21.15.90:50280) 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f151a000640 1 -- 172.21.15.90:0/2974769031 learned_addr learned my addr 172.21.15.90:0/2974769031 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1518c00640 1 --2- 172.21.15.90:0/2974769031 >> v2:172.21.15.102:3300/0 conn(0x7f15140b3d90 0x7f15140b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1519600640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15141449a0 0x7f151415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1513e00640 1 -- 172.21.15.90:0/2974769031 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946684224 0 0) 0x7f15140b47e0 con 0x7f151406b820 2024-03-31T15:55:02.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1513e00640 1 -- 172.21.15.90:0/2974769031 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14fc005740 con 0x7f151406b820 2024-03-31T15:55:02.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1518c00640 1 -- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15141449a0 msgr2=0x7f151415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:02.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1518c00640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15141449a0 0x7f151415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1518c00640 1 -- 172.21.15.90:0/2974769031 >> v1:172.21.15.90:6789/0 conn(0x7f151406b820 legacy=0x7f151415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.645+0000 7f1518c00640 1 -- 172.21.15.90:0/2974769031 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151415bab0 con 0x7f15140b3d90 2024-03-31T15:55:02.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f1518c00640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15140b3d90 0x7f15140b41f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f15000097c0 tx=0x7f151415f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbd1792bbb91797 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f1513e00640 1 -- 172.21.15.90:0/2974769031 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000aa620 con 0x7f15140b3d90 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f1513e00640 1 -- 172.21.15.90:0/2974769031 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15000af070 con 0x7f15140b3d90 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f1513e00640 1 -- 172.21.15.90:0/2974769031 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15000aa9c0 con 0x7f15140b3d90 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 -- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15140b3d90 msgr2=0x7f15140b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15140b3d90 0x7f15140b41f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f15000097c0 tx=0x7f151415f970 comp rx=0 tx=0).stop 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 -- 172.21.15.90:0/2974769031 shutdown_connections 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15140b3d90 0x7f15140b41f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 --2- 172.21.15.90:0/2974769031 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15141449a0 0x7f151415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 -- 172.21.15.90:0/2974769031 >> 172.21.15.90:0/2974769031 conn(0x7f15140b1c80 msgr2=0x7f15140b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:02.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 -- 172.21.15.90:0/2974769031 shutdown_connections 2024-03-31T15:55:02.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.649+0000 7f151c000640 1 -- 172.21.15.90:0/2974769031 wait complete. 2024-03-31T15:55:02.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 Processor -- start 2024-03-31T15:55:02.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- start start 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f151406b820 0x7f151415a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15140b3d90 0x7f151415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f15140b2e00 con 0x7f15141449a0 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f15140b2c80 con 0x7f151406b820 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f15140b2f80 con 0x7f15140b3d90 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41914/0 (socket says 172.21.15.90:41914) 2024-03-31T15:55:02.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 -- 172.21.15.90:0/1317161557 learned_addr learned my addr 172.21.15.90:0/1317161557 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f1518c00640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15140b3d90 0x7f151415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f1519600640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f151406b820 0x7f151415a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 -- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15140b3d90 msgr2=0x7f151415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15140b3d90 0x7f151415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 -- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f151406b820 msgr2=0x7f151415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f151406b820 0x7f151415a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 -- 172.21.15.90:0/1317161557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151415ad80 con 0x7f15141449a0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151a000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f150c00cb70 tx=0x7f150c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c017070 con 0x7f15141449a0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f150c00ad40 con 0x7f15141449a0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1514160030 con 0x7f15141449a0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c012620 con 0x7f15141449a0 2024-03-31T15:55:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.653+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15140cef30 con 0x7f15141449a0 2024-03-31T15:55:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15140686d0 con 0x7f15141449a0 2024-03-31T15:55:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f150c01a070 con 0x7f15141449a0 2024-03-31T15:55:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f1511600640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 0x7f14f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f150c086e80 con 0x7f15141449a0 2024-03-31T15:55:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f1519600640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 0x7f14f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.657+0000 7f1519600640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 0x7f14f406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1508009ac0 tx=0x7f1508009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.661+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f150c021020 con 0x7f15141449a0 2024-03-31T15:55:02.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.817+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15140cc930 con 0x7f15141449a0 2024-03-31T15:55:02.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f1511600640 1 -- 172.21.15.90:0/1317161557 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f150c056ad0 con 0x7f15141449a0 2024-03-31T15:55:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:02.813 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 msgr2=0x7f14f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 0x7f14f406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1508009ac0 tx=0x7f1508009190 comp rx=0 tx=0).stop 2024-03-31T15:55:02.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 msgr2=0x7f15140cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f150c00cb70 tx=0x7f150c00a430 comp rx=0 tx=0).stop 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 shutdown_connections 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f14f4068380 0x7f14f406a840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f15141449a0 0x7f15140cfcf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f15140b3d90 0x7f151415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 --2- 172.21.15.90:0/1317161557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f151406b820 0x7f151415a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 >> 172.21.15.90:0/1317161557 conn(0x7f15140b1c80 msgr2=0x7f151415ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 shutdown_connections 2024-03-31T15:55:02.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.821+0000 7f151c000640 1 -- 172.21.15.90:0/1317161557 wait complete. 2024-03-31T15:55:02.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:53:44.959463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:53:46.080667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-31T15:53:49.773692+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-31T15:55:02.845 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:02.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.989+0000 7f59be200640 1 Processor -- start 2024-03-31T15:55:02.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.989+0000 7f59be200640 1 -- start start 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59be200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b81616c0 0x7f59b80cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59be200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b80cb640 con 0x7f59b80afae0 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59be200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b80cb8e0 con 0x7f59b80ce830 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59be200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f59b80cc210 con 0x7f59b81616c0 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f59b81616c0 0x7f59b80cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bd800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f59b80afae0 0x7f59b80c61f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43060/0 (socket says 172.21.15.90:43060) 2024-03-31T15:55:02.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bd800640 1 -- 172.21.15.90:0/2569104015 learned_addr learned my addr 172.21.15.90:0/2569104015 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bce00640 1 -- 172.21.15.90:0/2569104015 >> v1:172.21.15.102:6789/0 conn(0x7f59b80ce830 legacy=0x7f59b80cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bce00640 1 -- 172.21.15.90:0/2569104015 >> v1:172.21.15.90:6789/0 conn(0x7f59b80afae0 legacy=0x7f59b80c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59bce00640 1 -- 172.21.15.90:0/2569104015 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b80cbbc0 con 0x7f59b81616c0 2024-03-31T15:55:02.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.993+0000 7f59b7400640 1 -- 172.21.15.90:0/2569104015 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175482931 0 0) 0x7f59b80cb640 con 0x7f59b80afae0 2024-03-31T15:55:02.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59bce00640 1 --2- 172.21.15.90:0/2569104015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b81616c0 0x7f59b80cf240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0097c0 tx=0x7f59ac01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3770619a886f0253 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59b7400640 1 -- 172.21.15.90:0/2569104015 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b0070 con 0x7f59b81616c0 2024-03-31T15:55:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59b7400640 1 -- 172.21.15.90:0/2569104015 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac0ab600 con 0x7f59b81616c0 2024-03-31T15:55:02.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59b7400640 1 -- 172.21.15.90:0/2569104015 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0ab9d0 con 0x7f59b81616c0 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- 172.21.15.90:0/2569104015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b81616c0 msgr2=0x7f59b80cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 --2- 172.21.15.90:0/2569104015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b81616c0 0x7f59b80cf240 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0097c0 tx=0x7f59ac01c410 comp rx=0 tx=0).stop 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- 172.21.15.90:0/2569104015 shutdown_connections 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 --2- 172.21.15.90:0/2569104015 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b81616c0 0x7f59b80cf240 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- 172.21.15.90:0/2569104015 >> 172.21.15.90:0/2569104015 conn(0x7f59b80d0c60 msgr2=0x7f59b80d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- 172.21.15.90:0/2569104015 shutdown_connections 2024-03-31T15:55:02.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- 172.21.15.90:0/2569104015 wait complete. 2024-03-31T15:55:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 Processor -- start 2024-03-31T15:55:02.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:02.997+0000 7f59be200640 1 -- start start 2024-03-31T15:55:02.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 0x7f59b80cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 0x7f59b8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8161c20 con 0x7f59b80afae0 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8161aa0 con 0x7f59b80ce830 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f59b816c680 con 0x7f59b81616c0 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34014/0 (socket says 172.21.15.90:34014) 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 -- 172.21.15.90:0/3115086642 learned_addr learned my addr 172.21.15.90:0/3115086642 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 0x7f59b80cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bd800640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 0x7f59b8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 -- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 msgr2=0x7f59b8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 0x7f59b8156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 -- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 msgr2=0x7f59b80cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 0x7f59b80cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 -- 172.21.15.90:0/3115086642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b80b5600 con 0x7f59b80afae0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 0x7f59b80cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bd800640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 0x7f59b8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59bce00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0097c0 tx=0x7f59ac01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0b0040 con 0x7f59b80afae0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ac004030 con 0x7f59b80afae0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac0043e0 con 0x7f59b80afae0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b80b58e0 con 0x7f59b80afae0 2024-03-31T15:55:02.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b80caa10 con 0x7f59b80afae0 2024-03-31T15:55:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b8068d00 con 0x7f59b80afae0 2024-03-31T15:55:02.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.001+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f59ac003620 con 0x7f59b80afae0 2024-03-31T15:55:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.005+0000 7f59b4c00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 0x7f599806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.005+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f59ac121b00 con 0x7f59b80afae0 2024-03-31T15:55:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.005+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 0x7f599806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:02.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.005+0000 7f59b7e00640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 0x7f599806a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f59b80aeaa0 tx=0x7f59a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.013+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f59ac016090 con 0x7f59b80afae0 2024-03-31T15:55:03.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.173+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59b80741d0 con 0x7f59b80afae0 2024-03-31T15:55:03.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.173+0000 7f59b4c00640 1 -- 172.21.15.90:0/3115086642 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f59ac121e20 con 0x7f59b80afae0 2024-03-31T15:55:03.167 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 msgr2=0x7f599806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 0x7f599806a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f59b80aeaa0 tx=0x7f59a4009040 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 msgr2=0x7f59b80b63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0097c0 tx=0x7f59ac01ac50 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 shutdown_connections 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5998068380 0x7f599806a840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f59b81616c0 0x7f59b8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59b80ce830 0x7f59b80cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 --2- 172.21.15.90:0/3115086642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59b80afae0 0x7f59b80b63c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 >> 172.21.15.90:0/3115086642 conn(0x7f59b80d0c60 msgr2=0x7f59b80b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:03.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 shutdown_connections 2024-03-31T15:55:03.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.177+0000 7f59be200640 1 -- 172.21.15.90:0/3115086642 wait complete. 2024-03-31T15:55:03.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:03.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 Processor -- start 2024-03-31T15:55:03.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 -- start start 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8e5006d020 0x7f8e500ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e500cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e5015e8c0 con 0x7f8e5015e4e0 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8e500d0ff0 con 0x7f8e5006d020 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e55800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8e500d12d0 con 0x7f8e5006d490 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8e5015e4e0 0x7f8e5006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43068/0 (socket says 172.21.15.90:43068) 2024-03-31T15:55:03.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4ea00640 1 -- 172.21.15.90:0/2277471621 learned_addr learned my addr 172.21.15.90:0/2277471621 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1138752081 0 0) 0x7f8e5015e8c0 con 0x7f8e5015e4e0 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e34003530 con 0x7f8e5015e4e0 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4e000640 1 --2- 172.21.15.90:0/2277471621 >> v2:172.21.15.102:3300/0 conn(0x7f8e5006d020 0x7f8e500ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4f400640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e500cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2980939522 0 0) 0x7f8e34003530 con 0x7f8e5015e4e0 2024-03-31T15:55:03.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 msgr2=0x7f8e500cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e500cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 >> v2:172.21.15.102:3300/0 conn(0x7f8e5006d020 msgr2=0x7f8e500ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 --2- 172.21.15.90:0/2277471621 >> v2:172.21.15.102:3300/0 conn(0x7f8e5006d020 0x7f8e500ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e500cc710 con 0x7f8e5015e4e0 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f8e40001680 con 0x7f8e5015e4e0 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4e000640 1 --2- 172.21.15.90:0/2277471621 >> v2:172.21.15.102:3300/0 conn(0x7f8e5006d020 0x7f8e500ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.333+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 >> v1:172.21.15.90:6789/0 conn(0x7f8e5015e4e0 legacy=0x7f8e5006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e34003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e34004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e34004d50 0x7f8e340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500d0ff0 con 0x7f8e340039c0 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500d12d0 con 0x7f8e34004d50 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3400bb30 con 0x7f8e5006d490 2024-03-31T15:55:03.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e34003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4f400640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e34004d50 0x7f8e340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4e000640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e34004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 -- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 msgr2=0x7f8e34004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5006d490 0x7f8e34004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 -- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e34004d50 msgr2=0x7f8e340051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e34004d50 0x7f8e340051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 -- 172.21.15.90:0/2277471621 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e34003530 con 0x7f8e340039c0 2024-03-31T15:55:03.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e34003de0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000c210 tx=0x7f8e4001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641e351e65eb639c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40007d40 con 0x7f8e340039c0 2024-03-31T15:55:03.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e400a9070 con 0x7f8e340039c0 2024-03-31T15:55:03.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.337+0000 7f8e4d600640 1 -- 172.21.15.90:0/2277471621 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40005d40 con 0x7f8e340039c0 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- 172.21.15.90:0/2277471621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 msgr2=0x7f8e34003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 --2- 172.21.15.90:0/2277471621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e34003de0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000c210 tx=0x7f8e4001cca0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4f400640 1 -- 172.21.15.90:0/2277471621 reap_dead start 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- 172.21.15.90:0/2277471621 shutdown_connections 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- 172.21.15.90:0/2277471621 >> 172.21.15.90:0/2277471621 conn(0x7f8e500d1f20 msgr2=0x7f8e500d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- 172.21.15.90:0/2277471621 shutdown_connections 2024-03-31T15:55:03.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- 172.21.15.90:0/2277471621 wait complete. 2024-03-31T15:55:03.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 Processor -- start 2024-03-31T15:55:03.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- start start 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5015b450 0x7f8e5015c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e5015b8b0 0x7f8e5015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500d0d90 con 0x7f8e340039c0 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8e500d0c10 con 0x7f8e5015b8b0 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e55800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8e500cdc60 con 0x7f8e5015b450 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34020/0 (socket says 172.21.15.90:34020) 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 -- 172.21.15.90:0/3573608345 learned_addr learned my addr 172.21.15.90:0/3573608345 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4e000640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5015b450 0x7f8e5015c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4f400640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e5015b8b0 0x7f8e5015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 -- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5015b450 msgr2=0x7f8e5015c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5015b450 0x7f8e5015c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 -- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e5015b8b0 msgr2=0x7f8e5015cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e5015b8b0 0x7f8e5015cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.341+0000 7f8e4ea00640 1 -- 172.21.15.90:0/3573608345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5015a110 con 0x7f8e340039c0 2024-03-31T15:55:03.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e4ea00640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c10 tx=0x7f8e34005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400047f0 con 0x7f8e340039c0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e40004990 con 0x7f8e340039c0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e40004d40 con 0x7f8e340039c0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5015a3b0 con 0x7f8e340039c0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5015a970 con 0x7f8e340039c0 2024-03-31T15:55:03.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e500685f0 con 0x7f8e340039c0 2024-03-31T15:55:03.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8e400b7030 con 0x7f8e340039c0 2024-03-31T15:55:03.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 0x7f8e3006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8e4011fd80 con 0x7f8e340039c0 2024-03-31T15:55:03.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.345+0000 7f8e4e000640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 0x7f8e3006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.349+0000 7f8e4e000640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 0x7f8e3006a870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e500d07a0 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.349+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8e400169b0 con 0x7f8e340039c0 2024-03-31T15:55:03.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8e50067700 con 0x7f8e340039c0 2024-03-31T15:55:03.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e46a00640 1 -- 172.21.15.90:0/3573608345 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f8e40007de0 con 0x7f8e340039c0 2024-03-31T15:55:03.501 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 msgr2=0x7f8e3006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 0x7f8e3006a870 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e500d07a0 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2024-03-31T15:55:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 msgr2=0x7f8e5014b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c10 tx=0x7f8e34005840 comp rx=0 tx=0).stop 2024-03-31T15:55:03.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 shutdown_connections 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8e300683b0 0x7f8e3006a870 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8e5015b8b0 0x7f8e5015cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8e5015b450 0x7f8e5015c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 --2- 172.21.15.90:0/3573608345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e340039c0 0x7f8e5014b0b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 >> 172.21.15.90:0/3573608345 conn(0x7f8e500d1f20 msgr2=0x7f8e501595e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 shutdown_connections 2024-03-31T15:55:03.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.509+0000 7f8e55800640 1 -- 172.21.15.90:0/3573608345 wait complete. 2024-03-31T15:55:03.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:03.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 Processor -- start 2024-03-31T15:55:03.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 -- start start 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5b580b0210 0x7f5b58166eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580cefc0 con 0x7f5b580d2b20 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b580cf2a0 con 0x7f5b5815c320 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b5ce00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5b580aed70 con 0x7f5b580b0210 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b4fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5b580d2b20 0x7f5b580cd100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43076/0 (socket says 172.21.15.90:43076) 2024-03-31T15:55:03.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b4fe00640 1 -- 172.21.15.90:0/2839957343 learned_addr learned my addr 172.21.15.90:0/2839957343 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:03.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b56800640 1 --2- 172.21.15.90:0/2839957343 >> v2:172.21.15.90:3300/0 conn(0x7f5b580b0210 0x7f5b58166eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b56800640 1 -- 172.21.15.90:0/2839957343 >> v1:172.21.15.102:6789/0 conn(0x7f5b5815c320 legacy=0x7f5b580cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b56800640 1 -- 172.21.15.90:0/2839957343 >> v1:172.21.15.90:6789/0 conn(0x7f5b580d2b20 legacy=0x7f5b580cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.669+0000 7f5b56800640 1 -- 172.21.15.90:0/2839957343 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b580cf580 con 0x7f5b580b0210 2024-03-31T15:55:03.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b56800640 1 --2- 172.21.15.90:0/2839957343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58166eb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009d90 tx=0x7f5b5001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3454ed0c15da6272 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b55400640 1 -- 172.21.15.90:0/2839957343 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500b2070 con 0x7f5b580b0210 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b55400640 1 -- 172.21.15.90:0/2839957343 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b500a58b0 con 0x7f5b580b0210 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b55400640 1 -- 172.21.15.90:0/2839957343 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500a5c80 con 0x7f5b580b0210 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 -- 172.21.15.90:0/2839957343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 msgr2=0x7f5b58166eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/2839957343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58166eb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b50009d90 tx=0x7f5b5001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 -- 172.21.15.90:0/2839957343 shutdown_connections 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/2839957343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58166eb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 -- 172.21.15.90:0/2839957343 >> 172.21.15.90:0/2839957343 conn(0x7f5b580d1f20 msgr2=0x7f5b580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 -- 172.21.15.90:0/2839957343 shutdown_connections 2024-03-31T15:55:03.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.673+0000 7f5b5ce00640 1 -- 172.21.15.90:0/2839957343 wait complete. 2024-03-31T15:55:03.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 Processor -- start 2024-03-31T15:55:03.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- start start 2024-03-31T15:55:03.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 0x7f5b58152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 0x7f5b580b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58169d90 con 0x7f5b580b0210 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58169c10 con 0x7f5b5815c320 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5b58169f10 con 0x7f5b580d2b20 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34024/0 (socket says 172.21.15.90:34024) 2024-03-31T15:55:03.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 -- 172.21.15.90:0/4161038249 learned_addr learned my addr 172.21.15.90:0/4161038249 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b56800640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 0x7f5b580b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b4fe00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 0x7f5b58152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 -- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 msgr2=0x7f5b58152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 0x7f5b58152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 -- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 msgr2=0x7f5b580b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 0x7f5b580b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 -- 172.21.15.90:0/4161038249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5815cbc0 con 0x7f5b580b0210 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b56800640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 0x7f5b580b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b4fe00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 0x7f5b58152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b55e00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800c500 tx=0x7f5b4800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48016020 con 0x7f5b580b0210 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b48005220 con 0x7f5b580b0210 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48007030 con 0x7f5b580b0210 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5815cea0 con 0x7f5b580b0210 2024-03-31T15:55:03.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5815d460 con 0x7f5b580b0210 2024-03-31T15:55:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.677+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b580685f0 con 0x7f5b580b0210 2024-03-31T15:55:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.681+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5b480071f0 con 0x7f5b580b0210 2024-03-31T15:55:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.681+0000 7f5b4e000640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 0x7f5b3406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.681+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5b48086820 con 0x7f5b580b0210 2024-03-31T15:55:03.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.681+0000 7f5b4fe00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 0x7f5b3406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:03.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.681+0000 7f5b4fe00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 0x7f5b3406a870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5b5815f970 tx=0x7f5b40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:03.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.689+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5b48020020 con 0x7f5b580b0210 2024-03-31T15:55:03.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.849+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5b580cd430 con 0x7f5b580b0210 2024-03-31T15:55:03.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.849+0000 7f5b4e000640 1 -- 172.21.15.90:0/4161038249 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f5b48056950 con 0x7f5b580b0210 2024-03-31T15:55:03.843 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:03.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 msgr2=0x7f5b3406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 0x7f5b3406a870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5b5815f970 tx=0x7f5b40009040 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 msgr2=0x7f5b58149ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5b4800c500 tx=0x7f5b4800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 shutdown_connections 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5b340683b0 0x7f5b3406a870 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5b5815f970 tx=0x7f5b40009040 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b5815c320 0x7f5b580b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5b580d2b20 0x7f5b58152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 --2- 172.21.15.90:0/4161038249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5b580b0210 0x7f5b58149ee0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 >> 172.21.15.90:0/4161038249 conn(0x7f5b580d1f20 msgr2=0x7f5b5806f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 shutdown_connections 2024-03-31T15:55:03.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:03.853+0000 7f5b5ce00640 1 -- 172.21.15.90:0/4161038249 wait complete. 2024-03-31T15:55:03.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-31T15:55:04.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.009+0000 7f81d5e00640 1 Processor -- start 2024-03-31T15:55:04.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.009+0000 7f81d5e00640 1 -- start start 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d01456c0 0x7f81d006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81d006bb30 0x7f81d00cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81d00cfde0 0x7f81d00721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f81d006bf90 con 0x7f81d00cfde0 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f81d006e7a0 con 0x7f81d006bb30 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f81d00d0590 con 0x7f81d01456c0 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81d00cfde0 0x7f81d00721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f81d00cfde0 0x7f81d00721f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34036/0 (socket says 172.21.15.90:34036) 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 -- 172.21.15.90:0/2985384340 learned_addr learned my addr 172.21.15.90:0/2985384340 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d4a00640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d01456c0 0x7f81d006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81cfe00640 1 --2- 172.21.15.90:0/2985384340 >> v2:172.21.15.102:3300/0 conn(0x7f81d006bb30 0x7f81d00cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 -- 172.21.15.90:0/2985384340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d01456c0 msgr2=0x7f81d006b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d01456c0 0x7f81d006b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 -- 172.21.15.90:0/2985384340 >> v2:172.21.15.102:3300/0 conn(0x7f81d006bb30 msgr2=0x7f81d00cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 --2- 172.21.15.90:0/2985384340 >> v2:172.21.15.102:3300/0 conn(0x7f81d006bb30 0x7f81d00cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.013+0000 7f81d5400640 1 -- 172.21.15.90:0/2985384340 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d00d0870 con 0x7f81d00cfde0 2024-03-31T15:55:04.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5400640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d00cfde0 0x7f81d00721f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f81c8009e50 tx=0x7f81c801c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6d06e7420f43e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81cf400640 1 -- 172.21.15.90:0/2985384340 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c80a6500 con 0x7f81d00cfde0 2024-03-31T15:55:04.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81cf400640 1 -- 172.21.15.90:0/2985384340 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c80a66a0 con 0x7f81d00cfde0 2024-03-31T15:55:04.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81cf400640 1 -- 172.21.15.90:0/2985384340 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c80a6a70 con 0x7f81d00cfde0 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- 172.21.15.90:0/2985384340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d00cfde0 msgr2=0x7f81d00721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d00cfde0 0x7f81d00721f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f81c8009e50 tx=0x7f81c801c670 comp rx=0 tx=0).stop 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- 172.21.15.90:0/2985384340 shutdown_connections 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d00cfde0 0x7f81d00721f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 --2- 172.21.15.90:0/2985384340 >> v2:172.21.15.102:3300/0 conn(0x7f81d006bb30 0x7f81d00cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 --2- 172.21.15.90:0/2985384340 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d01456c0 0x7f81d006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- 172.21.15.90:0/2985384340 >> 172.21.15.90:0/2985384340 conn(0x7f81d00d0c60 msgr2=0x7f81d00d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- 172.21.15.90:0/2985384340 shutdown_connections 2024-03-31T15:55:04.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- 172.21.15.90:0/2985384340 wait complete. 2024-03-31T15:55:04.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 Processor -- start 2024-03-31T15:55:04.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.017+0000 7f81d5e00640 1 -- start start 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d006bb30 0x7f81d01fb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00cfde0 0x7f81d00b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f81d006ec10 con 0x7f81d01456c0 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f81d00d0270 con 0x7f81d00cfde0 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f81d006ed90 con 0x7f81d006bb30 2024-03-31T15:55:04.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34048/0 (socket says 172.21.15.90:34048) 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 -- 172.21.15.90:0/371974709 learned_addr learned my addr 172.21.15.90:0/371974709 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d4a00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d006bb30 0x7f81d01fb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81cfe00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00cfde0 0x7f81d00b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 -- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d006bb30 msgr2=0x7f81d01fb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d006bb30 0x7f81d01fb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 -- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00cfde0 msgr2=0x7f81d00b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00cfde0 0x7f81d00b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 -- 172.21.15.90:0/371974709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d015dcf0 con 0x7f81d01456c0 2024-03-31T15:55:04.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5400640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81c801cf10 tx=0x7f81c8019410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c80bf070 con 0x7f81d01456c0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c80090d0 con 0x7f81d01456c0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c80a6ab0 con 0x7f81d01456c0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d015dfd0 con 0x7f81d01456c0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d015e590 con 0x7f81d01456c0 2024-03-31T15:55:04.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d00cce90 con 0x7f81d01456c0 2024-03-31T15:55:04.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.021+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f81c80a6c50 con 0x7f81d01456c0 2024-03-31T15:55:04.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.025+0000 7f81ccc00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 0x7f81b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.025+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f81c81208c0 con 0x7f81d01456c0 2024-03-31T15:55:04.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.025+0000 7f81d4a00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 0x7f81b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.025+0000 7f81d4a00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 0x7f81b006a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009a40 tx=0x7f81c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.029+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f81c80165e0 con 0x7f81d01456c0 2024-03-31T15:55:04.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f81d006f7f0 con 0x7f81d01456c0 2024-03-31T15:55:04.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81ccc00640 1 -- 172.21.15.90:0/371974709 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f81c80f0420 con 0x7f81d01456c0 2024-03-31T15:55:04.181 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 msgr2=0x7f81b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 0x7f81b006a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009a40 tx=0x7f81c4009190 comp rx=0 tx=0).stop 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 msgr2=0x7f81d013b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f81c801cf10 tx=0x7f81c8019410 comp rx=0 tx=0).stop 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 shutdown_connections 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81b0068380 0x7f81b006a840 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f81d01456c0 0x7f81d013b430 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00cfde0 0x7f81d00b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 --2- 172.21.15.90:0/371974709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f81d006bb30 0x7f81d01fb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 >> 172.21.15.90:0/371974709 conn(0x7f81d00d0c60 msgr2=0x7f81d00cd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 shutdown_connections 2024-03-31T15:55:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.189+0000 7f81d5e00640 1 -- 172.21.15.90:0/371974709 wait complete. 2024-03-31T15:55:04.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.353+0000 7fca89800640 1 Processor -- start 2024-03-31T15:55:04.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.353+0000 7fca89800640 1 -- start start 2024-03-31T15:55:04.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca89800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca840b1210 0x7fca840ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca89800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca840cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca89800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8415cc50 con 0x7fca8415c160 2024-03-31T15:55:04.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca89800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fca840cf1d0 con 0x7fca840b1210 2024-03-31T15:55:04.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca89800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fca8415cef0 con 0x7fca840ced60 2024-03-31T15:55:04.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fca8415c160 0x7fca8415c540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43084/0 (socket says 172.21.15.90:43084) 2024-03-31T15:55:04.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82a00640 1 -- 172.21.15.90:0/3885696968 learned_addr learned my addr 172.21.15.90:0/3885696968 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> v2:172.21.15.102:3300/0 conn(0x7fca840b1210 0x7fca840ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca83400640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca840cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165814330 0 0) 0x7fca8415cc50 con 0x7fca8415c160 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca68003530 con 0x7fca8415c160 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3052142741 0 0) 0x7fca68003530 con 0x7fca8415c160 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 msgr2=0x7fca840cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca840cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 >> v2:172.21.15.102:3300/0 conn(0x7fca840b1210 msgr2=0x7fca840ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 --2- 172.21.15.90:0/3885696968 >> v2:172.21.15.102:3300/0 conn(0x7fca840b1210 0x7fca840ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8415d1d0 con 0x7fca8415c160 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> v2:172.21.15.102:3300/0 conn(0x7fca840b1210 0x7fca840ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fca740016a0 con 0x7fca8415c160 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 >> v1:172.21.15.90:6789/0 conn(0x7fca8415c160 legacy=0x7fca8415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca68004810 0x7fca68004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca840cf1d0 con 0x7fca68004810 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca8415cef0 con 0x7fca680039c0 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fca6800b9d0 con 0x7fca840ced60 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca68004810 0x7fca68004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca83400640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82a00640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 -- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 msgr2=0x7fca680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca840ced60 0x7fca680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 -- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca680039c0 msgr2=0x7fca68003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca680039c0 0x7fca68003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.357+0000 7fca82000640 1 -- 172.21.15.90:0/3885696968 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca68003530 con 0x7fca68004810 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca82000640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca68004810 0x7fca68004c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fca70009cb0 tx=0x7fca7001cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb74edb1059aa0b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca700b2070 con 0x7fca68004810 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca700a5870 con 0x7fca68004810 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca81600640 1 -- 172.21.15.90:0/3885696968 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca70009880 con 0x7fca68004810 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- 172.21.15.90:0/3885696968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca68004810 msgr2=0x7fca68004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 --2- 172.21.15.90:0/3885696968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca68004810 0x7fca68004c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fca70009cb0 tx=0x7fca7001cc90 comp rx=0 tx=0).stop 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca83400640 1 -- 172.21.15.90:0/3885696968 reap_dead start 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- 172.21.15.90:0/3885696968 shutdown_connections 2024-03-31T15:55:04.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- 172.21.15.90:0/3885696968 >> 172.21.15.90:0/3885696968 conn(0x7fca840d1ed0 msgr2=0x7fca840d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- 172.21.15.90:0/3885696968 shutdown_connections 2024-03-31T15:55:04.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- 172.21.15.90:0/3885696968 wait complete. 2024-03-31T15:55:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 Processor -- start 2024-03-31T15:55:04.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.361+0000 7fca89800640 1 -- start start 2024-03-31T15:55:04.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68004810 0x7fca841e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca841f8220 0x7fca84152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca840cca50 con 0x7fca8406c700 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca8415fef0 con 0x7fca68004810 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fca840ccbd0 con 0x7fca841f8220 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca82a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68004810 0x7fca841e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34068/0 (socket says 172.21.15.90:34068) 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca82000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca841f8220 0x7fca84152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 -- 172.21.15.90:0/2509692116 learned_addr learned my addr 172.21.15.90:0/2509692116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 -- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca841f8220 msgr2=0x7fca84152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca841f8220 0x7fca84152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 -- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68004810 msgr2=0x7fca841e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68004810 0x7fca841e8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 -- 172.21.15.90:0/2509692116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca841e73c0 con 0x7fca8406c700 2024-03-31T15:55:04.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca83400640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fca7c011c80 tx=0x7fca7c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00fdb0 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7c01d070 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c015670 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca84152ab0 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca84153070 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.365+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca8406d0c0 con 0x7fca8406c700 2024-03-31T15:55:04.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.369+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fca7c015810 con 0x7fca8406c700 2024-03-31T15:55:04.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.369+0000 7fca7aa00640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 0x7fca6406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.369+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fca7c087520 con 0x7fca8406c700 2024-03-31T15:55:04.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.369+0000 7fca82a00640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 0x7fca6406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.369+0000 7fca82a00640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 0x7fca6406a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca84071d20 tx=0x7fca740014e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.373+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fca7c010040 con 0x7fca8406c700 2024-03-31T15:55:04.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.561+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca8415de00 con 0x7fca8406c700 2024-03-31T15:55:04.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.561+0000 7fca7aa00640 1 -- 172.21.15.90:0/2509692116 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fca7c057640 con 0x7fca8406c700 2024-03-31T15:55:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:04.555 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:55:04.556 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:55:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 msgr2=0x7fca6406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 0x7fca6406a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca84071d20 tx=0x7fca740014e0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 msgr2=0x7fca8406cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fca7c011c80 tx=0x7fca7c00f580 comp rx=0 tx=0).stop 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 shutdown_connections 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca64068380 0x7fca6406a840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca8406c700 0x7fca8406cb80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca841f8220 0x7fca84152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 --2- 172.21.15.90:0/2509692116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68004810 0x7fca841e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 >> 172.21.15.90:0/2509692116 conn(0x7fca840d1ed0 msgr2=0x7fca8415d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 shutdown_connections 2024-03-31T15:55:04.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.565+0000 7fca89800640 1 -- 172.21.15.90:0/2509692116 wait complete. 2024-03-31T15:55:04.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 Processor -- start 2024-03-31T15:55:04.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 -- start start 2024-03-31T15:55:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 0x7fc02015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0200b47e0 con 0x7fc0201449a0 2024-03-31T15:55:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc0200b4ac0 con 0x7fc02006bd60 2024-03-31T15:55:04.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc026e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc02015b7d0 con 0x7fc02015bfd0 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc024e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 0x7fc02015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc01fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc0201449a0 0x7fc0200b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:43364/0 (socket says 172.21.15.90:43364) 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc024e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 0x7fc02015ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:55192/0 (socket says 172.21.15.90:55192) 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc01fe00640 1 -- 172.21.15.90:0/185654617 learned_addr learned my addr 172.21.15.90:0/185654617 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:33154/0 (socket says 172.21.15.90:33154) 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 -- 172.21.15.90:0/185654617 >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 msgr2=0x7fc02015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 --2- 172.21.15.90:0/185654617 >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 0x7fc02015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 -- 172.21.15.90:0/185654617 >> v1:172.21.15.102:6789/0 conn(0x7fc0201449a0 legacy=0x7fc0200b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.721+0000 7fc017e00640 1 -- 172.21.15.90:0/185654617 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02015bab0 con 0x7fc02015bfd0 2024-03-31T15:55:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc017e00640 1 --2- 172.21.15.90:0/185654617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0080097c0 tx=0x7fc00801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5007a039095d65d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc01f400640 1 -- 172.21.15.90:0/185654617 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0080b0070 con 0x7fc02015bfd0 2024-03-31T15:55:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc01f400640 1 -- 172.21.15.90:0/185654617 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0080ab600 con 0x7fc02015bfd0 2024-03-31T15:55:04.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc01f400640 1 -- 172.21.15.90:0/185654617 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0080ab9d0 con 0x7fc02015bfd0 2024-03-31T15:55:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 -- 172.21.15.90:0/185654617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 msgr2=0x7fc02006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 --2- 172.21.15.90:0/185654617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0080097c0 tx=0x7fc00801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 -- 172.21.15.90:0/185654617 shutdown_connections 2024-03-31T15:55:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 --2- 172.21.15.90:0/185654617 >> v2:172.21.15.102:3300/0 conn(0x7fc02006bd60 0x7fc02015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 --2- 172.21.15.90:0/185654617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc02015bfd0 0x7fc02006b820 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 -- 172.21.15.90:0/185654617 >> 172.21.15.90:0/185654617 conn(0x7fc0200b1c80 msgr2=0x7fc0200b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 -- 172.21.15.90:0/185654617 shutdown_connections 2024-03-31T15:55:04.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.725+0000 7fc026e00640 1 -- 172.21.15.90:0/185654617 wait complete. 2024-03-31T15:55:04.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 Processor -- start 2024-03-31T15:55:04.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- start start 2024-03-31T15:55:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc0201449a0 0x7fc02015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 0x7fc0200ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc0200b2e00 con 0x7fc02006bd60 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc0200b2c80 con 0x7fc02015bfd0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc0200b2f80 con 0x7fc0201449a0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34078/0 (socket says 172.21.15.90:34078) 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 -- 172.21.15.90:0/2446471938 learned_addr learned my addr 172.21.15.90:0/2446471938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc017e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc0201449a0 0x7fc02015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc024e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 0x7fc0200ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 -- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc0201449a0 msgr2=0x7fc02015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc0201449a0 0x7fc02015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 -- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 msgr2=0x7fc0200ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 0x7fc0200ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 -- 172.21.15.90:0/2446471938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0200cf050 con 0x7fc02006bd60 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc024e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 0x7fc0200ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01fe00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c450 tx=0x7fc01000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010017070 con 0x7fc02006bd60 2024-03-31T15:55:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc010005cf0 con 0x7fc02006bd60 2024-03-31T15:55:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc010007290 con 0x7fc02006bd60 2024-03-31T15:55:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0200cf330 con 0x7fc02006bd60 2024-03-31T15:55:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.729+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0200cf8f0 con 0x7fc02006bd60 2024-03-31T15:55:04.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.733+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0200686d0 con 0x7fc02006bd60 2024-03-31T15:55:04.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.733+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc01001a070 con 0x7fc02006bd60 2024-03-31T15:55:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.733+0000 7fc01cc00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 0x7fbffc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.733+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc010086030 con 0x7fc02006bd60 2024-03-31T15:55:04.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.733+0000 7fc017e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 0x7fbffc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:04.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.737+0000 7fc017e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 0x7fbffc06a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ad000 tx=0x7fc0080a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:04.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.737+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc010055ba0 con 0x7fc02006bd60 2024-03-31T15:55:04.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.897+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc020072050 con 0x7fc02006bd60 2024-03-31T15:55:04.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.897+0000 7fc01cc00640 1 -- 172.21.15.90:0/2446471938 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7fc010059ea0 con 0x7fc02006bd60 2024-03-31T15:55:04.892 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:04.892 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 msgr2=0x7fbffc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 0x7fbffc06a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ad000 tx=0x7fc0080a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 msgr2=0x7fc020160140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c450 tx=0x7fc01000c910 comp rx=0 tx=0).stop 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 shutdown_connections 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbffc068380 0x7fbffc06a840 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.894 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc02015bfd0 0x7fc0200ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc0201449a0 0x7fc02015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 --2- 172.21.15.90:0/2446471938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc02006bd60 0x7fc020160140 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:04.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 >> 172.21.15.90:0/2446471938 conn(0x7fc0200b1c80 msgr2=0x7fc0200aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:04.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 shutdown_connections 2024-03-31T15:55:04.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:04.901+0000 7fc026e00640 1 -- 172.21.15.90:0/2446471938 wait complete. 2024-03-31T15:55:04.914 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:55:04.914 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:05.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 Processor -- start 2024-03-31T15:55:05.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 -- start start 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dec15beb0 0x7f8dec15f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8dec06bf30 0x7f8dec0715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dec162170 con 0x7f8dec06edb0 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8dec162450 con 0x7f8dec06bf30 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df4400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8dec162730 con 0x7f8dec15beb0 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8dec06bf30 0x7f8dec0715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df1000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8dec06edb0 0x7f8dec06b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43090/0 (socket says 172.21.15.90:43090) 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8dec06bf30 0x7f8dec0715e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34090/0 (socket says 172.21.15.90:34090) 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df1000640 1 -- 172.21.15.90:0/2198964405 learned_addr learned my addr 172.21.15.90:0/2198964405 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df1a00640 1 --2- 172.21.15.90:0/2198964405 >> v2:172.21.15.102:3300/0 conn(0x7f8dec15beb0 0x7f8dec15f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 -- 172.21.15.90:0/2198964405 >> v2:172.21.15.102:3300/0 conn(0x7f8dec15beb0 msgr2=0x7f8dec15f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 --2- 172.21.15.90:0/2198964405 >> v2:172.21.15.102:3300/0 conn(0x7f8dec15beb0 0x7f8dec15f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 -- 172.21.15.90:0/2198964405 >> v1:172.21.15.90:6789/0 conn(0x7f8dec06edb0 legacy=0x7f8dec06b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8df2400640 1 -- 172.21.15.90:0/2198964405 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec162a10 con 0x7f8dec06bf30 2024-03-31T15:55:05.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.057+0000 7f8debe00640 1 -- 172.21.15.90:0/2198964405 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062927442 0 0) 0x7f8dec162170 con 0x7f8dec06edb0 2024-03-31T15:55:05.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df2400640 1 --2- 172.21.15.90:0/2198964405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06bf30 0x7f8dec0715e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0aef50 tx=0x7f8de401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26adcf841f6f3321 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8debe00640 1 -- 172.21.15.90:0/2198964405 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de40a66e0 con 0x7f8dec06bf30 2024-03-31T15:55:05.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8debe00640 1 -- 172.21.15.90:0/2198964405 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de40a6880 con 0x7f8dec06bf30 2024-03-31T15:55:05.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8debe00640 1 -- 172.21.15.90:0/2198964405 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de40a6c50 con 0x7f8dec06bf30 2024-03-31T15:55:05.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 -- 172.21.15.90:0/2198964405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06bf30 msgr2=0x7f8dec0715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 --2- 172.21.15.90:0/2198964405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06bf30 0x7f8dec0715e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0aef50 tx=0x7f8de401c810 comp rx=0 tx=0).stop 2024-03-31T15:55:05.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 -- 172.21.15.90:0/2198964405 shutdown_connections 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 --2- 172.21.15.90:0/2198964405 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06bf30 0x7f8dec0715e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 --2- 172.21.15.90:0/2198964405 >> v2:172.21.15.102:3300/0 conn(0x7f8dec15beb0 0x7f8dec15f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 -- 172.21.15.90:0/2198964405 >> 172.21.15.90:0/2198964405 conn(0x7f8dec0b1c80 msgr2=0x7f8dec0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 -- 172.21.15.90:0/2198964405 shutdown_connections 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.061+0000 7f8df4400640 1 -- 172.21.15.90:0/2198964405 wait complete. 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 Processor -- start 2024-03-31T15:55:05.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- start start 2024-03-31T15:55:05.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8dec06bf30 0x7f8dec15a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 0x7f8dec15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec16b320 con 0x7f8dec06edb0 2024-03-31T15:55:05.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec16b1a0 con 0x7f8dec15beb0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8dec16b4a0 con 0x7f8dec06bf30 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34106/0 (socket says 172.21.15.90:34106) 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 -- 172.21.15.90:0/958201374 learned_addr learned my addr 172.21.15.90:0/958201374 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1a00640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8dec06bf30 0x7f8dec15a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df2400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 0x7f8dec15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 -- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8dec06bf30 msgr2=0x7f8dec15a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8dec06bf30 0x7f8dec15a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 -- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 msgr2=0x7f8dec15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 0x7f8dec15f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 -- 172.21.15.90:0/958201374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec0cca10 con 0x7f8dec06edb0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df2400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 0x7f8dec15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df1000640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8007fd0 tx=0x7f8dd800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8017070 con 0x7f8dec06edb0 2024-03-31T15:55:05.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd8005d10 con 0x7f8dec06edb0 2024-03-31T15:55:05.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8007310 con 0x7f8dec06edb0 2024-03-31T15:55:05.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec0cccf0 con 0x7f8dec06edb0 2024-03-31T15:55:05.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dec0cd2b0 con 0x7f8dec06edb0 2024-03-31T15:55:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.065+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dec0686d0 con 0x7f8dec06edb0 2024-03-31T15:55:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.069+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8dd801a070 con 0x7f8dec06edb0 2024-03-31T15:55:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.069+0000 7f8de9600640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 0x7f8dcc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.069+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8dd80861c0 con 0x7f8dec06edb0 2024-03-31T15:55:05.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.069+0000 7f8df1a00640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 0x7f8dcc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.069+0000 7f8df1a00640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 0x7f8dcc06a870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8dec161d00 tx=0x7f8de0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.073+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8dd8056370 con 0x7f8dec06edb0 2024-03-31T15:55:05.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.265+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dec074390 con 0x7f8dec06edb0 2024-03-31T15:55:05.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.265+0000 7f8de9600640 1 -- 172.21.15.90:0/958201374 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7f8dd805c070 con 0x7f8dec06edb0 2024-03-31T15:55:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:05.260 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:55:05.261 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:55:05.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 msgr2=0x7f8dcc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 0x7f8dcc06a870 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8dec161d00 tx=0x7f8de0008040 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 msgr2=0x7f8dec15eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8007fd0 tx=0x7f8dd800c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 shutdown_connections 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dcc0683b0 0x7f8dcc06a870 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dec15beb0 0x7f8dec15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8dec06edb0 0x7f8dec15eff0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 --2- 172.21.15.90:0/958201374 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8dec06bf30 0x7f8dec15a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 >> 172.21.15.90:0/958201374 conn(0x7f8dec0b1c80 msgr2=0x7f8dec0ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 shutdown_connections 2024-03-31T15:55:05.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.269+0000 7f8df4400640 1 -- 172.21.15.90:0/958201374 wait complete. 2024-03-31T15:55:05.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:05.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 Processor -- start 2024-03-31T15:55:05.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 -- start start 2024-03-31T15:55:05.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 0x7fb184162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb18406d7c0 0x7fb18415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb18415f780 con 0x7fb18406c7b0 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb1841622b0 con 0x7fb18406d7c0 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb18a400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb18415f1c0 con 0x7fb18415e1b0 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb17be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb18406c7b0 0x7fb18406d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43100/0 (socket says 172.21.15.90:43100) 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb17be00640 1 -- 172.21.15.90:0/1693617322 learned_addr learned my addr 172.21.15.90:0/1693617322 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb183400640 1 --2- 172.21.15.90:0/1693617322 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 0x7fb184162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb183e00640 1 --2- 172.21.15.90:0/1693617322 >> v2:172.21.15.102:3300/0 conn(0x7fb18406d7c0 0x7fb18415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.425+0000 7fb182a00640 1 -- 172.21.15.90:0/1693617322 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1117812270 0 0) 0x7fb18415f780 con 0x7fb18406c7b0 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb182a00640 1 -- 172.21.15.90:0/1693617322 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb168003530 con 0x7fb18406c7b0 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb183400640 1 -- 172.21.15.90:0/1693617322 >> v2:172.21.15.102:3300/0 conn(0x7fb18406d7c0 msgr2=0x7fb18415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb183400640 1 --2- 172.21.15.90:0/1693617322 >> v2:172.21.15.102:3300/0 conn(0x7fb18406d7c0 0x7fb18415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb183400640 1 -- 172.21.15.90:0/1693617322 >> v1:172.21.15.90:6789/0 conn(0x7fb18406c7b0 legacy=0x7fb18406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb183400640 1 -- 172.21.15.90:0/1693617322 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18415f4a0 con 0x7fb18415e1b0 2024-03-31T15:55:05.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb183400640 1 --2- 172.21.15.90:0/1693617322 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 0x7fb184162740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb1740097c0 tx=0x7fb17401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade0ed745c8a00b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb182a00640 1 -- 172.21.15.90:0/1693617322 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1740b1070 con 0x7fb18415e1b0 2024-03-31T15:55:05.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb182a00640 1 -- 172.21.15.90:0/1693617322 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1740a8050 con 0x7fb18415e1b0 2024-03-31T15:55:05.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb182a00640 1 -- 172.21.15.90:0/1693617322 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1740ac840 con 0x7fb18415e1b0 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 -- 172.21.15.90:0/1693617322 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 msgr2=0x7fb184162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 --2- 172.21.15.90:0/1693617322 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 0x7fb184162740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb1740097c0 tx=0x7fb17401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 -- 172.21.15.90:0/1693617322 shutdown_connections 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 --2- 172.21.15.90:0/1693617322 >> v2:172.21.15.102:3300/0 conn(0x7fb18406d7c0 0x7fb18415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 --2- 172.21.15.90:0/1693617322 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18415e1b0 0x7fb184162740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.429+0000 7fb18a400640 1 -- 172.21.15.90:0/1693617322 >> 172.21.15.90:0/1693617322 conn(0x7fb18409aa40 msgr2=0x7fb18409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- 172.21.15.90:0/1693617322 shutdown_connections 2024-03-31T15:55:05.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- 172.21.15.90:0/1693617322 wait complete. 2024-03-31T15:55:05.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 Processor -- start 2024-03-31T15:55:05.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- start start 2024-03-31T15:55:05.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18406c7b0 0x7fb184155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18406d7c0 0x7fb184155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb1840d2940 con 0x7fb18415e1b0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb1840d27c0 con 0x7fb18406d7c0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb18a400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb1840d2ac0 con 0x7fb18406c7b0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34120/0 (socket says 172.21.15.90:34120) 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 -- 172.21.15.90:0/2143367721 learned_addr learned my addr 172.21.15.90:0/2143367721 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18406c7b0 0x7fb184155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb17be00640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18406d7c0 0x7fb184155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 -- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18406c7b0 msgr2=0x7fb184155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18406c7b0 0x7fb184155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 -- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18406d7c0 msgr2=0x7fb184155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18406d7c0 0x7fb184155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 -- 172.21.15.90:0/2143367721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1840b5850 con 0x7fb18415e1b0 2024-03-31T15:55:05.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.433+0000 7fb183e00640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00efc0 tx=0x7fb17c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c018020 con 0x7fb18415e1b0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17c014480 con 0x7fb18415e1b0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1840b5b30 con 0x7fb18415e1b0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c014870 con 0x7fb18415e1b0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1840b63d0 con 0x7fb18415e1b0 2024-03-31T15:55:05.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1840686d0 con 0x7fb18415e1b0 2024-03-31T15:55:05.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb17c01b020 con 0x7fb18415e1b0 2024-03-31T15:55:05.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.437+0000 7fb17b400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 0x7fb16006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.441+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb17c087bc0 con 0x7fb18415e1b0 2024-03-31T15:55:05.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.441+0000 7fb183400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 0x7fb16006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.441+0000 7fb183400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 0x7fb16006a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb1740ae000 tx=0x7fb1740a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.445+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb17c022030 con 0x7fb18415e1b0 2024-03-31T15:55:05.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.633+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1840b65a0 con 0x7fb18415e1b0 2024-03-31T15:55:05.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.633+0000 7fb17b400640 1 -- 172.21.15.90:0/2143367721 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5177 (secure 0 0 0) 0x7fb17c010040 con 0x7fb18415e1b0 2024-03-31T15:55:05.627 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:05.627 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/928038037","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":928038037},{"type":"v1","addr":"172.21.15.102:6836","nonce":928038037}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4252667377","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4252667377},{"type":"v1","addr":"172.21.15.102:6837","nonce":4252667377}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/533164494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":533164494},{"type":"v1","addr":"172.21.15.90:6833","nonce":533164494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:53:47.115176+0000","modified":"2024-03-31T15:54:11.211162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/3620557044","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":3620557044},{"type":"v1","addr":"172.21.15.90:6835","nonce":3620557044}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-03-31T15:55:05.627 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 msgr2=0x7fb16006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 0x7fb16006a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb1740ae000 tx=0x7fb1740a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 msgr2=0x7fb184156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00efc0 tx=0x7fb17c00c490 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 shutdown_connections 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb160068380 0x7fb16006a840 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb18415e1b0 0x7fb184156530 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb18406d7c0 0x7fb184155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 --2- 172.21.15.90:0/2143367721 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb18406c7b0 0x7fb184155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 >> 172.21.15.90:0/2143367721 conn(0x7fb18409aa40 msgr2=0x7fb184070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 shutdown_connections 2024-03-31T15:55:05.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.637+0000 7fb18a400640 1 -- 172.21.15.90:0/2143367721 wait complete. 2024-03-31T15:55:05.648 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.789+0000 7f5cff200640 1 Processor -- start 2024-03-31T15:55:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.789+0000 7f5cff200640 1 -- start start 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cff200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5cf80d48d0 0x7f5cf806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cff200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf815e8a0 con 0x7f5cf815e4c0 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cff200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf806cf90 con 0x7f5cf806c7c0 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cff200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf806d270 con 0x7f5cf80d48d0 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5cf815e4c0 0x7f5cf80d41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43112/0 (socket says 172.21.15.90:43112) 2024-03-31T15:55:05.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7e00640 1 -- 172.21.15.90:0/2790989517 learned_addr learned my addr 172.21.15.90:0/2790989517 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2790989517 >> v2:172.21.15.102:3300/0 conn(0x7f5cf80d48d0 0x7f5cf806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365365421 0 0) 0x7f5cf815e8a0 con 0x7f5cf815e4c0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cdc003530 con 0x7f5cf815e4c0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3017407387 0 0) 0x7f5cdc003530 con 0x7f5cf815e4c0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 >> v2:172.21.15.102:3300/0 conn(0x7f5cf80d48d0 msgr2=0x7f5cf806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 --2- 172.21.15.90:0/2790989517 >> v2:172.21.15.102:3300/0 conn(0x7f5cf80d48d0 0x7f5cf806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 >> v1:172.21.15.102:6789/0 conn(0x7f5cf806c7c0 legacy=0x7f5cf80cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf806d550 con 0x7f5cf815e4c0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f5ce80016a0 con 0x7f5cf815e4c0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 >> v1:172.21.15.90:6789/0 conn(0x7f5cf815e4c0 legacy=0x7f5cf80d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0039c0 0x7f5cdc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cdc004810 0x7f5cdc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 0x7f5cdc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf806d270 con 0x7f5cdc0051d0 2024-03-31T15:55:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc00c020 con 0x7f5cdc0039c0 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5cdc00c220 con 0x7f5cdc004810 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 0x7f5cdc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cf7e00640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0039c0 0x7f5cdc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cdc004810 0x7f5cdc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 -- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cdc004810 msgr2=0x7f5cdc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cdc004810 0x7f5cdc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 -- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0039c0 msgr2=0x7f5cdc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0039c0 0x7f5cdc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cfd200640 1 -- 172.21.15.90:0/2790989517 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc003530 con 0x7f5cdc0051d0 2024-03-31T15:55:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.793+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cdc004810 0x7f5cdc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cfd200640 1 -- 172.21.15.90:0/2790989517 reap_dead start 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cfd200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 0x7f5cdc00b9d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0001f60 tx=0x7f5cf001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b8337a5d995a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0001b60 con 0x7f5cdc0051d0 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0003b80 con 0x7f5cdc0051d0 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cf7400640 1 -- 172.21.15.90:0/2790989517 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00050d0 con 0x7f5cdc0051d0 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 -- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 msgr2=0x7f5cdc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 0x7f5cdc00b9d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0001f60 tx=0x7f5cf001c880 comp rx=0 tx=0).stop 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 -- 172.21.15.90:0/2790989517 shutdown_connections 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 --2- 172.21.15.90:0/2790989517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cdc0051d0 0x7f5cdc00b9d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 -- 172.21.15.90:0/2790989517 >> 172.21.15.90:0/2790989517 conn(0x7f5cf808db60 msgr2=0x7f5cf809d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 -- 172.21.15.90:0/2790989517 shutdown_connections 2024-03-31T15:55:05.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.797+0000 7f5cff200640 1 -- 172.21.15.90:0/2790989517 wait complete. 2024-03-31T15:55:05.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 Processor -- start 2024-03-31T15:55:05.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- start start 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0051d0 0x7f5cf815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cf80b79b0 0x7f5cf80b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf806cbe0 con 0x7f5cf80b6d80 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf80ceef0 con 0x7f5cdc0051d0 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf806cd60 con 0x7f5cf80b79b0 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34144/0 (socket says 172.21.15.90:34144) 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 -- 172.21.15.90:0/2490379191 learned_addr learned my addr 172.21.15.90:0/2490379191 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:05.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cfd200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cf80b79b0 0x7f5cf80b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cf7e00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0051d0 0x7f5cf815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 -- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cf80b79b0 msgr2=0x7f5cf80b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cf80b79b0 0x7f5cf80b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 -- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0051d0 msgr2=0x7f5cf815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0051d0 0x7f5cf815c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 -- 172.21.15.90:0/2490379191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf81f3e70 con 0x7f5cf80b6d80 2024-03-31T15:55:05.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cefe00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009160 tx=0x7f5ce00119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0004220 con 0x7f5cf80b6d80 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce00043c0 con 0x7f5cf80b6d80 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0015890 con 0x7f5cf80b6d80 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf81f4bf0 con 0x7f5cf80b6d80 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf81f4110 con 0x7f5cf80b6d80 2024-03-31T15:55:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.801+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf80686d0 con 0x7f5cf80b6d80 2024-03-31T15:55:05.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.805+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ce0015a30 con 0x7f5cf80b6d80 2024-03-31T15:55:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.805+0000 7f5cf4c00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 0x7f5cd406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.805+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5ce0087280 con 0x7f5cf80b6d80 2024-03-31T15:55:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.805+0000 7f5cf7e00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 0x7f5cd406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:05.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.805+0000 7f5cf7e00640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 0x7f5cd406a840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8001810 tx=0x7f5ce80014e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:05.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.809+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5ce0019030 con 0x7f5cf80b6d80 2024-03-31T15:55:05.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.969+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5cf806cf40 con 0x7f5cf80b6d80 2024-03-31T15:55:05.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cf4c00640 1 -- 172.21.15.90:0/2490379191 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18866 (secure 0 0 0) 0x7f5ce0056df0 con 0x7f5cf80b6d80 2024-03-31T15:55:05.965 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:05.965 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:54:30.934954+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:53:44.959463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-31T15:53:46.080667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-31T15:53:49.773692+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 msgr2=0x7f5cd406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 0x7f5cd406a840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8001810 tx=0x7f5ce80014e0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 msgr2=0x7f5cf80b7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5ce0009160 tx=0x7f5ce00119a0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 shutdown_connections 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5cd4068380 0x7f5cd406a840 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cf80b79b0 0x7f5cf80b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cf80b6d80 0x7f5cf80b7200 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 --2- 172.21.15.90:0/2490379191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc0051d0 0x7f5cf815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:05.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 >> 172.21.15.90:0/2490379191 conn(0x7f5cf808db60 msgr2=0x7f5cf80d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:05.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 shutdown_connections 2024-03-31T15:55:05.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:05.973+0000 7f5cff200640 1 -- 172.21.15.90:0/2490379191 wait complete. 2024-03-31T15:55:05.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:55:06.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 Processor -- start 2024-03-31T15:55:06.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 -- start start 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 0x7f29840d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29840d2dc0 con 0x7f29840d2940 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29840d1400 con 0x7f29841624a0 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f298a600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f298415e700 con 0x7f298415e320 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f29840d2940 0x7f2984161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43122/0 (socket says 172.21.15.90:43122) 2024-03-31T15:55:06.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982a00640 1 -- 172.21.15.90:0/3505486199 learned_addr learned my addr 172.21.15.90:0/3505486199 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2983400640 1 --2- 172.21.15.90:0/3505486199 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 0x7f29840d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3715208563 0 0) 0x7f29840d2dc0 con 0x7f29840d2940 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2968003530 con 0x7f29840d2940 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2817536374 0 0) 0x7f29840d1400 con 0x7f29841624a0 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29840d2dc0 con 0x7f29841624a0 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2983400640 1 -- 172.21.15.90:0/3505486199 >> v1:172.21.15.102:6789/0 conn(0x7f29841624a0 legacy=0x7f29840b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2983400640 1 -- 172.21.15.90:0/3505486199 >> v1:172.21.15.90:6789/0 conn(0x7f29840d2940 legacy=0x7f2984161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1796091063 0 0) 0x7f2968003530 con 0x7f29840d2940 2024-03-31T15:55:06.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.133+0000 7f2983400640 1 -- 172.21.15.90:0/3505486199 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29840b1a20 con 0x7f298415e320 2024-03-31T15:55:06.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f2983400640 1 --2- 172.21.15.90:0/3505486199 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 0x7f29840d0d20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29740097f0 tx=0x7f297401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d06d585e35aa47 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29740b1070 con 0x7f298415e320 2024-03-31T15:55:06.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29740ac400 con 0x7f298415e320 2024-03-31T15:55:06.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f2982000640 1 -- 172.21.15.90:0/3505486199 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970001680 con 0x7f298415e320 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 -- 172.21.15.90:0/3505486199 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 msgr2=0x7f29840d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 --2- 172.21.15.90:0/3505486199 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 0x7f29840d0d20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29740097f0 tx=0x7f297401c400 comp rx=0 tx=0).stop 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 -- 172.21.15.90:0/3505486199 shutdown_connections 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 --2- 172.21.15.90:0/3505486199 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f298415e320 0x7f29840d0d20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 -- 172.21.15.90:0/3505486199 >> 172.21.15.90:0/3505486199 conn(0x7f298408bfd0 msgr2=0x7f29840d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 -- 172.21.15.90:0/3505486199 shutdown_connections 2024-03-31T15:55:06.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.137+0000 7f298a600640 1 -- 172.21.15.90:0/3505486199 wait complete. 2024-03-31T15:55:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 Processor -- start 2024-03-31T15:55:06.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- start start 2024-03-31T15:55:06.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 0x7f29840ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29841624a0 0x7f2984154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29840b37b0 con 0x7f29840d2940 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f29840b3630 con 0x7f298415e320 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f298416bf10 con 0x7f29841624a0 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34148/0 (socket says 172.21.15.90:34148) 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 -- 172.21.15.90:0/3770878666 learned_addr learned my addr 172.21.15.90:0/3770878666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2982a00640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 0x7f29840ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983e00640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29841624a0 0x7f2984154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 -- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29841624a0 msgr2=0x7f2984154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29841624a0 0x7f2984154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 -- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 msgr2=0x7f29840ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 0x7f29840ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 -- 172.21.15.90:0/3770878666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29840cf090 con 0x7f29840d2940 2024-03-31T15:55:06.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2982a00640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 0x7f29840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f2983400640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c10 tx=0x7f29740191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29740b1040 con 0x7f29840d2940 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29740036e0 con 0x7f29840d2940 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29840cd620 con 0x7f29840d2940 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974003a90 con 0x7f29840d2940 2024-03-31T15:55:06.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.141+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29840cdbe0 con 0x7f29840d2940 2024-03-31T15:55:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2984068660 con 0x7f29840d2940 2024-03-31T15:55:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29740be030 con 0x7f29840d2940 2024-03-31T15:55:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f297b400640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 0x7f296406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f297411ffc0 con 0x7f29840d2940 2024-03-31T15:55:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f2982a00640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 0x7f296406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.145+0000 7f2982a00640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 0x7f296406a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2970000fe0 tx=0x7f2970009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.149+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f29740c5020 con 0x7f29840d2940 2024-03-31T15:55:06.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.477+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f29840b26a0 con 0x7f29840d2940 2024-03-31T15:55:06.472 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-31T15:55:06.481+0000 7fc737200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:55:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.481+0000 7f297b400640 1 -- 172.21.15.90:0/3770878666 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f2974015ff0 con 0x7f29840d2940 2024-03-31T15:55:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:55:06.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 msgr2=0x7f296406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 0x7f296406a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2970000fe0 tx=0x7f2970009040 comp rx=0 tx=0).stop 2024-03-31T15:55:06.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 msgr2=0x7f29840cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c10 tx=0x7f29740191a0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 shutdown_connections 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2964068380 0x7f296406a840 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29841624a0 0x7f2984154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f298415e320 0x7f29840ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 --2- 172.21.15.90:0/3770878666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29840d2940 0x7f29840cd020 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 >> 172.21.15.90:0/3770878666 conn(0x7f298408bfd0 msgr2=0x7f29840d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 shutdown_connections 2024-03-31T15:55:06.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.485+0000 7f298a600640 1 -- 172.21.15.90:0/3770878666 wait complete. 2024-03-31T15:55:06.487 INFO:tasks.ceph.mds.a.smithi090.stderr: -5046> 2024-03-31T15:53:53.391+0000 7fc431c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:55:06.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:55:06.535 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-31T15:55:06.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 Processor -- start 2024-03-31T15:55:06.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 -- start start 2024-03-31T15:55:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fc015c320 0x7f2fc00cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 0x7f2fc00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc00cd470 0x7f2fc00cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc00aed70 con 0x7f2fc00cd470 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc00cd900 con 0x7f2fc015c320 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fc4c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc00cefc0 con 0x7f2fc00d2b20 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc00cd470 0x7f2fc00cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc00cd470 0x7f2fc00cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34176/0 (socket says 172.21.15.90:34176) 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 -- 172.21.15.90:0/3799166157 learned_addr learned my addr 172.21.15.90:0/3799166157 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbde00640 1 --2- 172.21.15.90:0/3799166157 >> v2:172.21.15.102:3300/0 conn(0x7f2fc015c320 0x7f2fc00cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fb5400640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 0x7f2fc00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 -- 172.21.15.90:0/3799166157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 msgr2=0x7f2fc00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 0x7f2fc00ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 -- 172.21.15.90:0/3799166157 >> v2:172.21.15.102:3300/0 conn(0x7f2fc015c320 msgr2=0x7f2fc00cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:06.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 --2- 172.21.15.90:0/3799166157 >> v2:172.21.15.102:3300/0 conn(0x7f2fc015c320 0x7f2fc00cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fbe800640 1 -- 172.21.15.90:0/3799166157 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc00cf2a0 con 0x7f2fc00cd470 2024-03-31T15:55:06.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.645+0000 7f2fb5400640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 0x7f2fc00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:06.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fbe800640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc00cd470 0x7f2fc00cfd00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00739b0 tx=0x7f2fb801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a67195a23e0847 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fbd400640 1 -- 172.21.15.90:0/3799166157 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80b0070 con 0x7f2fc00cd470 2024-03-31T15:55:06.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fbd400640 1 -- 172.21.15.90:0/3799166157 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb80ab600 con 0x7f2fc00cd470 2024-03-31T15:55:06.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fbd400640 1 -- 172.21.15.90:0/3799166157 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80ab9d0 con 0x7f2fc00cd470 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 -- 172.21.15.90:0/3799166157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc00cd470 msgr2=0x7f2fc00cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc00cd470 0x7f2fc00cfd00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fc00739b0 tx=0x7f2fb801c490 comp rx=0 tx=0).stop 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 -- 172.21.15.90:0/3799166157 shutdown_connections 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc00cd470 0x7f2fc00cfd00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/3799166157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00d2b20 0x7f2fc00ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/3799166157 >> v2:172.21.15.102:3300/0 conn(0x7f2fc015c320 0x7f2fc00cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 -- 172.21.15.90:0/3799166157 >> 172.21.15.90:0/3799166157 conn(0x7f2fc00d1f20 msgr2=0x7f2fc00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 -- 172.21.15.90:0/3799166157 shutdown_connections 2024-03-31T15:55:06.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.649+0000 7f2fc4c00640 1 -- 172.21.15.90:0/3799166157 wait complete. 2024-03-31T15:55:06.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 Processor -- start 2024-03-31T15:55:06.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- start start 2024-03-31T15:55:06.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00cd470 0x7f2fc014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc00d2b20 0x7f2fc006e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc00d0e20 con 0x7f2fc015c320 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc00d0ca0 con 0x7f2fc00d2b20 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc00af1e0 con 0x7f2fc00cd470 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34180/0 (socket says 172.21.15.90:34180) 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbde00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00cd470 0x7f2fc014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 -- 172.21.15.90:0/159467746 learned_addr learned my addr 172.21.15.90:0/159467746 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fb5400640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc00d2b20 0x7f2fc006e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 -- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00cd470 msgr2=0x7f2fc014f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00cd470 0x7f2fc014f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 -- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc00d2b20 msgr2=0x7f2fc006e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc00d2b20 0x7f2fc006e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:06.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 -- 172.21.15.90:0/159467746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc01643b0 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fbe800640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80ae040 tx=0x7f2fb8019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80b0040 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb80b9070 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc0164690 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80abe60 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.653+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc0164c50 con 0x7f2fc015c320 2024-03-31T15:55:06.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc00685f0 con 0x7f2fc015c320 2024-03-31T15:55:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2fb80b4640 con 0x7f2fc015c320 2024-03-31T15:55:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fb6a00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 0x7f2f9c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f2fb8120940 con 0x7f2fc015c320 2024-03-31T15:55:06.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fbde00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 0x7f2f9c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:06.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.657+0000 7f2fbde00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 0x7f2f9c06a870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0158c70 tx=0x7f2fb000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:06.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.661+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2fb8016110 con 0x7f2fc015c320 2024-03-31T15:55:06.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:06.857+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2fc006f620 con 0x7f2fc015c320 2024-03-31T15:55:07.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.485+0000 7f2fb6a00640 1 -- 172.21.15.90:0/159467746 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f2fb80f03b0 con 0x7f2fc015c320 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 msgr2=0x7f2f9c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 0x7f2f9c06a870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0158c70 tx=0x7f2fb000b040 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 msgr2=0x7f2fc00b1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80ae040 tx=0x7f2fb8019230 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 shutdown_connections 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2f9c0683b0 0x7f2f9c06a870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc015c320 0x7f2fc00b1fe0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc00d2b20 0x7f2fc006e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 --2- 172.21.15.90:0/159467746 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc00cd470 0x7f2fc014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 >> 172.21.15.90:0/159467746 conn(0x7f2fc00d1f20 msgr2=0x7f2fc00b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 shutdown_connections 2024-03-31T15:55:07.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.489+0000 7f2fc4c00640 1 -- 172.21.15.90:0/159467746 wait complete. 2024-03-31T15:55:07.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:55:07.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 Processor -- start 2024-03-31T15:55:07.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 -- start start 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1f0415b150 0x7f1f0406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f0415eba0 0x7f1f040ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f0415b530 con 0x7f1f040caf20 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f1f0415b850 con 0x7f1f0415b150 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f08c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0415f310 con 0x7f1f0415eba0 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f01e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1f0415b150 0x7f1f0406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:07.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f01e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f1f0415b150 0x7f1f0406eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34188/0 (socket says 172.21.15.90:34188) 2024-03-31T15:55:07.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.689+0000 7f1f01e00640 1 -- 172.21.15.90:0/2362239221 learned_addr learned my addr 172.21.15.90:0/2362239221 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:07.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01e00640 1 -- 172.21.15.90:0/2362239221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f0415eba0 msgr2=0x7f1f040ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:07.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01e00640 1 --2- 172.21.15.90:0/2362239221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f0415eba0 0x7f1f040ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01e00640 1 -- 172.21.15.90:0/2362239221 >> v1:172.21.15.102:6789/0 conn(0x7f1f040caf20 legacy=0x7f1f0415bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:07.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01e00640 1 -- 172.21.15.90:0/2362239221 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0415f5f0 con 0x7f1f0415b150 2024-03-31T15:55:07.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01e00640 1 --2- 172.21.15.90:0/2362239221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415b150 0x7f1f0406eac0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009870 tx=0x7f1ef401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc34bea8a4d931d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:07.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01400640 1 -- 172.21.15.90:0/2362239221 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40b0070 con 0x7f1f0415b150 2024-03-31T15:55:07.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01400640 1 -- 172.21.15.90:0/2362239221 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ef40ab600 con 0x7f1f0415b150 2024-03-31T15:55:07.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f01400640 1 -- 172.21.15.90:0/2362239221 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40ab9d0 con 0x7f1f0415b150 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 -- 172.21.15.90:0/2362239221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415b150 msgr2=0x7f1f0406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2362239221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415b150 0x7f1f0406eac0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009870 tx=0x7f1ef401c520 comp rx=0 tx=0).stop 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 -- 172.21.15.90:0/2362239221 shutdown_connections 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2362239221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f0415eba0 0x7f1f040ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2362239221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415b150 0x7f1f0406eac0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 -- 172.21.15.90:0/2362239221 >> 172.21.15.90:0/2362239221 conn(0x7f1f040b1880 msgr2=0x7f1f040b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:07.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 -- 172.21.15.90:0/2362239221 shutdown_connections 2024-03-31T15:55:07.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.693+0000 7f1f08c00640 1 -- 172.21.15.90:0/2362239221 wait complete. 2024-03-31T15:55:07.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 Processor -- start 2024-03-31T15:55:07.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- start start 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f040caf20 0x7f1f0415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f0415b150 0x7f1f040c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1f041680c0 con 0x7f1f0415eba0 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1f04167f40 con 0x7f1f0415b150 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1f04168240 con 0x7f1f040caf20 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34202/0 (socket says 172.21.15.90:34202) 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 -- 172.21.15.90:0/2998342545 learned_addr learned my addr 172.21.15.90:0/2998342545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f01e00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f040caf20 0x7f1f0415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:07.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1ef9400640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f0415b150 0x7f1f040c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 -- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f040caf20 msgr2=0x7f1f0415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f040caf20 0x7f1f0415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 -- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f0415b150 msgr2=0x7f1f040c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f0415b150 0x7f1f040c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 -- 172.21.15.90:0/2998342545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0415d040 con 0x7f1f0415eba0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f02800640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00c780 tx=0x7f1efc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc004b70 con 0x7f1f0415eba0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1efc004d10 con 0x7f1f0415eba0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00aa30 con 0x7f1f0415eba0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0415d320 con 0x7f1f0415eba0 2024-03-31T15:55:07.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.697+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0415d8e0 con 0x7f1f0415eba0 2024-03-31T15:55:07.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f040686d0 con 0x7f1f0415eba0 2024-03-31T15:55:07.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1efc00abd0 con 0x7f1f0415eba0 2024-03-31T15:55:07.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1efaa00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 0x7f1ee006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:07.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f1efc086fe0 con 0x7f1f0415eba0 2024-03-31T15:55:07.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1f01e00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 0x7f1ee006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:07.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.701+0000 7f1f01e00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 0x7f1ee006a870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40ad000 tx=0x7f1ef40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:07.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.705+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1efc020070 con 0x7f1f0415eba0 2024-03-31T15:55:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:07.865+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1f040cdf30 con 0x7f1f0415eba0 2024-03-31T15:55:08.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.489+0000 7f1efaa00640 1 -- 172.21.15.90:0/2998342545 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7f1efc056a70 con 0x7f1f0415eba0 2024-03-31T15:55:08.482 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 msgr2=0x7f1ee006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 0x7f1ee006a870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40ad000 tx=0x7f1ef40a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 msgr2=0x7f1f040c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00c780 tx=0x7f1efc00cc40 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 shutdown_connections 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ee00683b0 0x7f1ee006a870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1f0415eba0 0x7f1f040c7ed0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f0415b150 0x7f1f040c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 --2- 172.21.15.90:0/2998342545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1f040caf20 0x7f1f0415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 >> 172.21.15.90:0/2998342545 conn(0x7f1f040b1880 msgr2=0x7f1f040ca060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 shutdown_connections 2024-03-31T15:55:08.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.493+0000 7f1f08c00640 1 -- 172.21.15.90:0/2998342545 wait complete. 2024-03-31T15:55:08.509 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:55:08.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 Processor -- start 2024-03-31T15:55:08.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 -- start start 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f89c40b2bb0 0x7f89c40d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c406d5d0 con 0x7f89c40d0fa0 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c406d8b0 con 0x7f89c406d110 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c8c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f89c40d05d0 con 0x7f89c40b2bb0 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f89c40b2bb0 0x7f89c40d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f89c40b2bb0 0x7f89c40d0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34208/0 (socket says 172.21.15.90:34208) 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f89c40d0fa0 0x7f89c4161d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43142/0 (socket says 172.21.15.90:43142) 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 -- 172.21.15.90:0/1107161196 learned_addr learned my addr 172.21.15.90:0/1107161196 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 -- 172.21.15.90:0/1107161196 >> v1:172.21.15.102:6789/0 conn(0x7f89c406d110 legacy=0x7f89c40b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 -- 172.21.15.90:0/1107161196 >> v1:172.21.15.90:6789/0 conn(0x7f89c40d0fa0 legacy=0x7f89c4161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c3400640 1 -- 172.21.15.90:0/1107161196 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c40d0260 con 0x7f89c40b2bb0 2024-03-31T15:55:08.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.657+0000 7f89c2000640 1 -- 172.21.15.90:0/1107161196 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766996907 0 0) 0x7f89c406d5d0 con 0x7f89c40d0fa0 2024-03-31T15:55:08.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c3400640 1 --2- 172.21.15.90:0/1107161196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c40b2bb0 0x7f89c40d0a60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4971325fa780c009 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:08.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c2000640 1 -- 172.21.15.90:0/1107161196 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b40a58e0 con 0x7f89c40b2bb0 2024-03-31T15:55:08.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c2000640 1 -- 172.21.15.90:0/1107161196 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b40a5a80 con 0x7f89c40b2bb0 2024-03-31T15:55:08.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c2000640 1 -- 172.21.15.90:0/1107161196 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b40a5e50 con 0x7f89c40b2bb0 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 -- 172.21.15.90:0/1107161196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c40b2bb0 msgr2=0x7f89c40d0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 --2- 172.21.15.90:0/1107161196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c40b2bb0 0x7f89c40d0a60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 -- 172.21.15.90:0/1107161196 shutdown_connections 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 --2- 172.21.15.90:0/1107161196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c40b2bb0 0x7f89c40d0a60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 -- 172.21.15.90:0/1107161196 >> 172.21.15.90:0/1107161196 conn(0x7f89c40d1d70 msgr2=0x7f89c40d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 -- 172.21.15.90:0/1107161196 shutdown_connections 2024-03-31T15:55:08.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.661+0000 7f89c8c00640 1 -- 172.21.15.90:0/1107161196 wait complete. 2024-03-31T15:55:08.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 Processor -- start 2024-03-31T15:55:08.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- start start 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 0x7f89c40afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f89c40d0fa0 0x7f89c4159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f89c416be50 con 0x7f89c406d110 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f89c40b33a0 con 0x7f89c40b2bb0 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f89c416bfd0 con 0x7f89c40d0fa0 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34210/0 (socket says 172.21.15.90:34210) 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 -- 172.21.15.90:0/3108250061 learned_addr learned my addr 172.21.15.90:0/3108250061 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3e00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f89c40d0fa0 0x7f89c4159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c2a00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 0x7f89c40afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 -- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f89c40d0fa0 msgr2=0x7f89c4159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f89c40d0fa0 0x7f89c4159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 -- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 msgr2=0x7f89c40afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:08.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 0x7f89c40afbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:08.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 -- 172.21.15.90:0/3108250061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c4154340 con 0x7f89c406d110 2024-03-31T15:55:08.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c2a00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 0x7f89c40afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c3400640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f89b4019600 tx=0x7f89b4019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b4009510 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b40be070 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c415d7f0 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b40b86b0 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.665+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c415ddb0 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c40685f0 con 0x7f89c406d110 2024-03-31T15:55:08.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f89b40b8850 con 0x7f89c406d110 2024-03-31T15:55:08.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89bb400640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 0x7f89a406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:08.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7f89b4122640 con 0x7f89c406d110 2024-03-31T15:55:08.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89c2a00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 0x7f89a406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:08.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.669+0000 7f89c2a00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 0x7f89a406a870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89c4159290 tx=0x7f89b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:08.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.673+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f89b40165e0 con 0x7f89c406d110 2024-03-31T15:55:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:08.837+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f89c40b7bf0 con 0x7f89c406d110 2024-03-31T15:55:09.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.493+0000 7f89bb400640 1 -- 172.21.15.90:0/3108250061 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f89b40f28f0 con 0x7f89c406d110 2024-03-31T15:55:09.485 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 msgr2=0x7f89a406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 0x7f89a406a870 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f89c4159290 tx=0x7f89b0009040 comp rx=0 tx=0).stop 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 msgr2=0x7f89c4154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f89b4019600 tx=0x7f89b4019790 comp rx=0 tx=0).stop 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 shutdown_connections 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f89a40683b0 0x7f89a406a870 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f89c40d0fa0 0x7f89c4159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c40b2bb0 0x7f89c40afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 --2- 172.21.15.90:0/3108250061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f89c406d110 0x7f89c4154ce0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 >> 172.21.15.90:0/3108250061 conn(0x7f89c40d1d70 msgr2=0x7f89c40ccf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 shutdown_connections 2024-03-31T15:55:09.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.497+0000 7f89c8c00640 1 -- 172.21.15.90:0/3108250061 wait complete. 2024-03-31T15:55:09.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-31T15:55:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.657+0000 7fd0e2600640 1 Processor -- start 2024-03-31T15:55:09.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.657+0000 7fd0e2600640 1 -- start start 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0e2600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0dc15b150 0x7fd0dc06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0e2600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd0dc15bce0 0x7fd0dc15c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0e2600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0dc15b530 con 0x7fd0dc15eba0 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0e2600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc15b850 con 0x7fd0dc15bce0 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0e2600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd0dc15f310 con 0x7fd0dc15b150 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0dbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd0dc15bce0 0x7fd0dc15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0d2a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd0dc15eba0 0x7fd0dc0ca9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43158/0 (socket says 172.21.15.90:43158) 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0d2a00640 1 -- 172.21.15.90:0/247354978 learned_addr learned my addr 172.21.15.90:0/247354978 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0db400640 1 --2- 172.21.15.90:0/247354978 >> v2:172.21.15.102:3300/0 conn(0x7fd0dc15b150 0x7fd0dc06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0dbe00640 1 -- 172.21.15.90:0/247354978 >> v2:172.21.15.102:3300/0 conn(0x7fd0dc15b150 msgr2=0x7fd0dc06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:09.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0dbe00640 1 --2- 172.21.15.90:0/247354978 >> v2:172.21.15.102:3300/0 conn(0x7fd0dc15b150 0x7fd0dc06eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0dbe00640 1 -- 172.21.15.90:0/247354978 >> v1:172.21.15.90:6789/0 conn(0x7fd0dc15eba0 legacy=0x7fd0dc0ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0dbe00640 1 -- 172.21.15.90:0/247354978 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc15f5f0 con 0x7fd0dc15bce0 2024-03-31T15:55:09.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.661+0000 7fd0daa00640 1 -- 172.21.15.90:0/247354978 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525099854 0 0) 0x7fd0dc15b530 con 0x7fd0dc15eba0 2024-03-31T15:55:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0dbe00640 1 --2- 172.21.15.90:0/247354978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc15c110 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0b28e0 tx=0x7fd0d401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ade9063683b743 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0daa00640 1 -- 172.21.15.90:0/247354978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40b2070 con 0x7fd0dc15bce0 2024-03-31T15:55:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0daa00640 1 -- 172.21.15.90:0/247354978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d40a5900 con 0x7fd0dc15bce0 2024-03-31T15:55:09.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0daa00640 1 -- 172.21.15.90:0/247354978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40a5cd0 con 0x7fd0dc15bce0 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- 172.21.15.90:0/247354978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15bce0 msgr2=0x7fd0dc15c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 --2- 172.21.15.90:0/247354978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc15c110 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0b28e0 tx=0x7fd0d401c930 comp rx=0 tx=0).stop 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- 172.21.15.90:0/247354978 shutdown_connections 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 --2- 172.21.15.90:0/247354978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc15c110 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 --2- 172.21.15.90:0/247354978 >> v2:172.21.15.102:3300/0 conn(0x7fd0dc15b150 0x7fd0dc06eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- 172.21.15.90:0/247354978 >> 172.21.15.90:0/247354978 conn(0x7fd0dc0b1880 msgr2=0x7fd0dc0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- 172.21.15.90:0/247354978 shutdown_connections 2024-03-31T15:55:09.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- 172.21.15.90:0/247354978 wait complete. 2024-03-31T15:55:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 Processor -- start 2024-03-31T15:55:09.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.665+0000 7fd0e2600640 1 -- start start 2024-03-31T15:55:09.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0dc15b150 0x7fd0dc0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc1680c0 con 0x7fd0dc15eba0 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc167f40 con 0x7fd0dc15bce0 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd0dc168240 con 0x7fd0dc15b150 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34226/0 (socket says 172.21.15.90:34226) 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 -- 172.21.15.90:0/198921489 learned_addr learned my addr 172.21.15.90:0/198921489 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d2a00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0db400640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0dc15b150 0x7fd0dc0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 -- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0dc15b150 msgr2=0x7fd0dc0c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0dc15b150 0x7fd0dc0c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 -- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 msgr2=0x7fd0dc152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 -- 172.21.15.90:0/198921489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc0c66e0 con 0x7fd0dc15eba0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d2a00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0dbe00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40b0040 tx=0x7fd0d40196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40b2050 con 0x7fd0dc15eba0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d40be070 con 0x7fd0dc15eba0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40ada90 con 0x7fd0dc15eba0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc0c44a0 con 0x7fd0dc15eba0 2024-03-31T15:55:09.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc0c4740 con 0x7fd0dc15eba0 2024-03-31T15:55:09.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc0686d0 con 0x7fd0dc15eba0 2024-03-31T15:55:09.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.669+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd0d40adc30 con 0x7fd0dc15eba0 2024-03-31T15:55:09.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.673+0000 7fd0d3e00640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 0x7fd0b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:09.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.673+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7fd0d4121bf0 con 0x7fd0dc15eba0 2024-03-31T15:55:09.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.673+0000 7fd0db400640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 0x7fd0b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:09.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.673+0000 7fd0db400640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 0x7fd0b806a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0c8b50 tx=0x7fd0cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:09.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.677+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd0d4016520 con 0x7fd0dc15eba0 2024-03-31T15:55:09.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:09.833+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd0dc0cdf30 con 0x7fd0dc15eba0 2024-03-31T15:55:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.497+0000 7fd0d3e00640 1 -- 172.21.15.90:0/198921489 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7fd0d40ab340 con 0x7fd0dc15eba0 2024-03-31T15:55:10.492 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data_ec' removed 2024-03-31T15:55:10.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 msgr2=0x7fd0b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 0x7fd0b806a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0c8b50 tx=0x7fd0cc008040 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 msgr2=0x7fd0dc0c3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd0d40b0040 tx=0x7fd0d40196d0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 shutdown_connections 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd0b8068380 0x7fd0b806a840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0dc15eba0 0x7fd0dc0c3840 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0dc15bce0 0x7fd0dc152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 --2- 172.21.15.90:0/198921489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0dc15b150 0x7fd0dc0c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 >> 172.21.15.90:0/198921489 conn(0x7fd0dc0b1880 msgr2=0x7fd0dc06d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.501+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 shutdown_connections 2024-03-31T15:55:10.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.505+0000 7fd0e2600640 1 -- 172.21.15.90:0/198921489 wait complete. 2024-03-31T15:55:10.541 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:55:10.542 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:55:10.542 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:55:10.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:10.542 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:55:10.542 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:55:10.542 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:55:10.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:10.543 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:55:10.543 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:55:10.543 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:55:10.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:10.543 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:55:10.543 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:55:10.543 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:55:10.544 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:10.555 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:55:10.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:55:10.556 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:55:10.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:55:10.557 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:55:10.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:55:10.560 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:55:10.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:55:10.603 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:55:10.603 DEBUG:teuthology.parallel:result is None 2024-03-31T15:55:10.604 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:55:10.604 DEBUG:teuthology.parallel:result is None 2024-03-31T15:55:10.607 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:55:10.608 DEBUG:teuthology.parallel:result is None 2024-03-31T15:55:10.610 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:55:10.610 DEBUG:teuthology.parallel:result is None 2024-03-31T15:55:10.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:55:10.688 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-31T15:55:10.688 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-31T15:55:10.697 INFO:tasks.ceph.mds.d.smithi102.stdout:starting mds.d at 2024-03-31T15:55:10.697 INFO:tasks.ceph.mds.b.smithi102.stdout:starting mds.b at 2024-03-31T15:55:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 Processor -- start 2024-03-31T15:55:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 -- start start 2024-03-31T15:55:10.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 0x7f8c0006d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:10.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c000d02c0 con 0x7f8c000d1080 2024-03-31T15:55:10.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c000d05a0 con 0x7f8c0015e070 2024-03-31T15:55:10.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.761+0000 7f8c04c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8c0006ce10 con 0x7f8c000cf400 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bff400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8c0015e070 0x7f8c0006c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:43410/0 (socket says 172.21.15.90:43410) 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bf6a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 0x7f8c0006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bff400640 1 -- 172.21.15.90:0/259500638 learned_addr learned my addr 172.21.15.90:0/259500638 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bffe00640 1 --1- 172.21.15.90:0/259500638 >> v1:172.21.15.90:6789/0 conn(0x7f8c000d1080 0x7f8c000cd610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43170/0 (socket says 172.21.15.90:43170) 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bfea00640 1 -- 172.21.15.90:0/259500638 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333018655 0 0) 0x7f8c000d05a0 con 0x7f8c0015e070 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bf6a00640 1 -- 172.21.15.90:0/259500638 >> v1:172.21.15.102:6789/0 conn(0x7f8c0015e070 legacy=0x7f8c0006c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bf6a00640 1 -- 172.21.15.90:0/259500638 >> v1:172.21.15.90:6789/0 conn(0x7f8c000d1080 legacy=0x7f8c000cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bf6a00640 1 -- 172.21.15.90:0/259500638 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c000d0880 con 0x7f8c000cf400 2024-03-31T15:55:10.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bf6a00640 1 --2- 172.21.15.90:0/259500638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 0x7f8c0006d110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8be80097f0 tx=0x7f8be801c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef69364c8b369f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:10.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bfea00640 1 -- 172.21.15.90:0/259500638 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be80b1070 con 0x7f8c000cf400 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bfea00640 1 -- 172.21.15.90:0/259500638 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be80ac640 con 0x7f8c000cf400 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.765+0000 7f8bfea00640 1 -- 172.21.15.90:0/259500638 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be80aca10 con 0x7f8c000cf400 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- 172.21.15.90:0/259500638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 msgr2=0x7f8c0006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 --2- 172.21.15.90:0/259500638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 0x7f8c0006d110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8be80097f0 tx=0x7f8be801c370 comp rx=0 tx=0).stop 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- 172.21.15.90:0/259500638 shutdown_connections 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 --2- 172.21.15.90:0/259500638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000cf400 0x7f8c0006d110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- 172.21.15.90:0/259500638 >> 172.21.15.90:0/259500638 conn(0x7f8c000d1d50 msgr2=0x7f8c000d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- 172.21.15.90:0/259500638 shutdown_connections 2024-03-31T15:55:10.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- 172.21.15.90:0/259500638 wait complete. 2024-03-31T15:55:10.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 Processor -- start 2024-03-31T15:55:10.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- start start 2024-03-31T15:55:10.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:10.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000d1080 0x7f8c0014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:10.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 0x7f8c000ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0015e450 con 0x7f8c000cf400 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c000d0bf0 con 0x7f8c0015e070 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8c04c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8c0015e5d0 con 0x7f8c000d1080 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34258/0 (socket says 172.21.15.90:34258) 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 -- 172.21.15.90:0/1448683964 learned_addr learned my addr 172.21.15.90:0/1448683964 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bffe00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 0x7f8c000ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bf6a00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000d1080 0x7f8c0014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 -- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000d1080 msgr2=0x7f8c0014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000d1080 0x7f8c0014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 -- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 msgr2=0x7f8c000ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 0x7f8c000ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:10.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bff400640 1 -- 172.21.15.90:0/1448683964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c000caba0 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.769+0000 7f8bffe00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 0x7f8c000ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bff400640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000c4a0 tx=0x7f8bf000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0018070 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf00074e0 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0004a30 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c000cae80 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c000cb440 con 0x7f8c000cf400 2024-03-31T15:55:10.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c000685f0 con 0x7f8c000cf400 2024-03-31T15:55:10.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8bf0004bd0 con 0x7f8c000cf400 2024-03-31T15:55:10.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 0x7f8bdc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:10.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f8bf0085ae0 con 0x7f8c000cf400 2024-03-31T15:55:10.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.773+0000 7f8bf6a00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 0x7f8bdc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:10.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.777+0000 7f8bf6a00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 0x7f8bdc06a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8be80ae000 tx=0x7f8be8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:10.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.777+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8bf0055a90 con 0x7f8c000cf400 2024-03-31T15:55:10.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:10.945+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f8c00070e60 con 0x7f8c000cf400 2024-03-31T15:55:11.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.509+0000 7f8bf7e00640 1 -- 172.21.15.90:0/1448683964 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f8bf0059d90 con 0x7f8c000cf400 2024-03-31T15:55:11.502 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 msgr2=0x7f8bdc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 0x7f8bdc06a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8be80ae000 tx=0x7f8be8008040 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 msgr2=0x7f8c00159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000c4a0 tx=0x7f8bf000c960 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 shutdown_connections 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8bdc0683b0 0x7f8bdc06a870 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c0015e070 0x7f8c000ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c000d1080 0x7f8c0014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 --2- 172.21.15.90:0/1448683964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c000cf400 0x7f8c00159180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 >> 172.21.15.90:0/1448683964 conn(0x7f8c000d1d50 msgr2=0x7f8c0015e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:11.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 shutdown_connections 2024-03-31T15:55:11.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.513+0000 7f8c04c00640 1 -- 172.21.15.90:0/1448683964 wait complete. 2024-03-31T15:55:11.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:55:11.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 Processor -- start 2024-03-31T15:55:11.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 -- start start 2024-03-31T15:55:11.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c2c1449a0 0x7f4c2c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c2c0b3d90 0x7f4c2c06b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2c0b47e0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2c0b4ac0 con 0x7f4c2c1449a0 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.677+0000 7f4c31400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2c15b7d0 con 0x7f4c2c0b3d90 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c2c1449a0 0x7f4c2c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2b200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4c2c0b4220 0x7f4c2c15ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43178/0 (socket says 172.21.15.90:43178) 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2b200640 1 -- 172.21.15.90:0/1822577588 learned_addr learned my addr 172.21.15.90:0/1822577588 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c29e00640 1 --2- 172.21.15.90:0/1822577588 >> v2:172.21.15.102:3300/0 conn(0x7f4c2c0b3d90 0x7f4c2c06b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 -- 172.21.15.90:0/1822577588 >> v2:172.21.15.102:3300/0 conn(0x7f4c2c0b3d90 msgr2=0x7f4c2c06b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1822577588 >> v2:172.21.15.102:3300/0 conn(0x7f4c2c0b3d90 0x7f4c2c06b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 -- 172.21.15.90:0/1822577588 >> v1:172.21.15.90:6789/0 conn(0x7f4c2c0b4220 legacy=0x7f4c2c15ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 -- 172.21.15.90:0/1822577588 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c15bab0 con 0x7f4c2c1449a0 2024-03-31T15:55:11.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c29400640 1 -- 172.21.15.90:0/1822577588 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286097189 0 0) 0x7f4c2c0b47e0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1822577588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c1449a0 0x7f4c2c15bfd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0b1810 tx=0x7f4c2c15f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e85ebbc5a57faf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:11.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c29400640 1 -- 172.21.15.90:0/1822577588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c0b1070 con 0x7f4c2c1449a0 2024-03-31T15:55:11.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c29400640 1 -- 172.21.15.90:0/1822577588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c0ac5f0 con 0x7f4c2c1449a0 2024-03-31T15:55:11.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.681+0000 7f4c29400640 1 -- 172.21.15.90:0/1822577588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c0ac9c0 con 0x7f4c2c1449a0 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1822577588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c1449a0 msgr2=0x7f4c2c15bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- 172.21.15.90:0/1822577588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c1449a0 0x7f4c2c15bfd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0b1810 tx=0x7f4c2c15f970 comp rx=0 tx=0).stop 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1822577588 shutdown_connections 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- 172.21.15.90:0/1822577588 >> v2:172.21.15.102:3300/0 conn(0x7f4c2c0b3d90 0x7f4c2c06b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- 172.21.15.90:0/1822577588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c1449a0 0x7f4c2c15bfd0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1822577588 >> 172.21.15.90:0/1822577588 conn(0x7f4c2c0b1c80 msgr2=0x7f4c2c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1822577588 shutdown_connections 2024-03-31T15:55:11.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1822577588 wait complete. 2024-03-31T15:55:11.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 Processor -- start 2024-03-31T15:55:11.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- start start 2024-03-31T15:55:11.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 0x7f4c2c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.680 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 0x7f4c2c1fa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c168000 con 0x7f4c2c0b4220 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c06c170 con 0x7f4c2c0b3d90 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2c168180 con 0x7f4c2c1449a0 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34284/0 (socket says 172.21.15.90:34284) 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 -- 172.21.15.90:0/1355261984 learned_addr learned my addr 172.21.15.90:0/1355261984 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c2b200640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 0x7f4c2c1fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 -- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 msgr2=0x7f4c2c1fa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 0x7f4c2c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 0x7f4c2c1fa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 -- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 msgr2=0x7f4c2c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 0x7f4c2c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 -- 172.21.15.90:0/1355261984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c15cda0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c2b200640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 0x7f4c2c1fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 0x7f4c2c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c29e00640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e8a0 tx=0x7f4c1400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14017070 con 0x7f4c2c0b4220 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1400c940 con 0x7f4c2c0b4220 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1400ccf0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.685+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c15d080 con 0x7f4c2c0b4220 2024-03-31T15:55:11.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2c15d640 con 0x7f4c2c0b4220 2024-03-31T15:55:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c2c0686d0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4c1401a070 con 0x7f4c2c0b4220 2024-03-31T15:55:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c22a00640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 0x7f4c0806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f4c14086a80 con 0x7f4c2c0b4220 2024-03-31T15:55:11.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 0x7f4c0806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:11.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.689+0000 7f4c2a800640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 0x7f4c0806a840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0ae000 tx=0x7f4c1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:11.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.693+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4c14088e20 con 0x7f4c2c0b4220 2024-03-31T15:55:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.905+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4c2c072050 con 0x7f4c2c0b4220 2024-03-31T15:55:11.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.905+0000 7f4c22a00640 1 -- 172.21.15.90:0/1355261984 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f4c140571f0 con 0x7f4c2c0b4220 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmcninMRAARhope6BpTuawcm/Bd/MSIg==", 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmli3zARAAHVKo3x7gWq9M18P3MNqxXA==", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-03-31T15:55:11.900 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmcqabOBAAqFr5SZkMULMCOz06c7V0qw==", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglm+6ITCRAACtqVZ3FpKoRXagsrFx3ywg==", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:55:11.901 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDVhglm8T+UJxAA3z+5rswh0d7RakAvfO7J8g==", 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.902 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDYhglmke27LhAAF0oYalYC+cMjqc4Iw1r4kg==", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDbhglmcDWmNBAAuAJakIes6OBv3yzAhtT0DA==", 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.903 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDehglm7Y42OxAA8+5GF4XF+3cTnxhi2BhiRw==", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDlhglmJwDhARAAzCpZr8/syvLjfnI0qi4jpA==", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDohglmGXmMBxAAQKoVfHNdxRWZIjtF3TN8CA==", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.905 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDrhglmjsnMDBAAYOGRa9UwgXIIjj6hbg8fwg==", 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDuhglmcWRFERAAWDwc/7Xlku2oXqBELqBCPg==", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A==", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.907 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmSpJzFhAAdwvdZGC6Tfafcb5kqX9/Zw==", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmDJHvEhAA/EJD8+c4iK8u7ZEroAHp9g==", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglm2+uRHBAABJCOXHPmeC+WUichEcsMHw==", 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.909 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmNbnzFRAAdnmwvGW9dCDLKNHHXy/8BA==", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:55:11.910 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mds", 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglmNfqHChAAaFPW82/+LNzP6yIwmD4rWg==", 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mds" 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-mgr", 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglmMgSIChAAG27Mb6Zx9cuYT8duCCKXMg==", 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-osd", 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglmywyIChAABUuFIup56AAP3j0b8np7jA==", 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-osd" 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd", 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglm9xSIChAASpb1hSqzFCfq3gvWzLzIUg==", 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglmlh2IChAAM7li8AM/iP8KfMMHSEVLRw==", 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.bootstrap-rgw", 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQD0hglmMyaIChAAyd8jzTMMNzKjHm0HKuPljw==", 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmdRVMLRAAHrLHygDnOeGTKMmMCVyzpg==", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmy8KFKRAAyHCJHySxd3Jcb8lQBIA7eA==", 2024-03-31T15:55:11.914 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-31T15:55:11.915 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 msgr2=0x7f4c0806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 0x7f4c0806a840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0ae000 tx=0x7f4c1c008040 comp rx=0 tx=0).stop 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 msgr2=0x7f4c2c1fa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400e8a0 tx=0x7f4c1400eca0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 shutdown_connections 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c08068380 0x7f4c0806a840 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c2c1449a0 0x7f4c2c1fa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c2c0b4220 0x7f4c2c1fa3f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 --2- 172.21.15.90:0/1355261984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c2c0b3d90 0x7f4c2c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 >> 172.21.15.90:0/1355261984 conn(0x7f4c2c0b1c80 msgr2=0x7f4c2c141780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 shutdown_connections 2024-03-31T15:55:11.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:11.909+0000 7f4c31400640 1 -- 172.21.15.90:0/1355261984 wait complete. 2024-03-31T15:55:11.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-31T15:55:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 Processor -- start 2024-03-31T15:55:12.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 -- start start 2024-03-31T15:55:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d04151d70 0x7f1d040cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 0x7f1d0415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d04071cd0 con 0x7f1d04072a90 2024-03-31T15:55:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1d04071fb0 con 0x7f1d04151d70 2024-03-31T15:55:12.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.061+0000 7f1d0b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d04072290 con 0x7f1d04152150 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d09400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 0x7f1d0415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d04151d70 0x7f1d040cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1cfbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1d04072a90 0x7f1d0415efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:43424/0 (socket says 172.21.15.90:43424) 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1cfbe00640 1 -- 172.21.15.90:0/3347553011 learned_addr learned my addr 172.21.15.90:0/3347553011 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 -- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 msgr2=0x7f1d0415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 0x7f1d0415f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 -- 172.21.15.90:0/3347553011 >> v1:172.21.15.102:6789/0 conn(0x7f1d04072a90 legacy=0x7f1d0415efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 -- 172.21.15.90:0/3347553011 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d04072570 con 0x7f1d04151d70 2024-03-31T15:55:12.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d09400640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 0x7f1d0415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:12.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d08a00640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04151d70 0x7f1d040cc440 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40097c0 tx=0x7f1cf401cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27b27b11ce923393 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.065+0000 7f1d03e00640 1 -- 172.21.15.90:0/3347553011 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40aa620 con 0x7f1d04151d70 2024-03-31T15:55:12.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d03e00640 1 -- 172.21.15.90:0/3347553011 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cf40af070 con 0x7f1d04151d70 2024-03-31T15:55:12.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d03e00640 1 -- 172.21.15.90:0/3347553011 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40aa9e0 con 0x7f1d04151d70 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04151d70 msgr2=0x7f1d040cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04151d70 0x7f1d040cc440 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40097c0 tx=0x7f1cf401cf00 comp rx=0 tx=0).stop 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- 172.21.15.90:0/3347553011 shutdown_connections 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04152150 0x7f1d0415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- 172.21.15.90:0/3347553011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04151d70 0x7f1d040cc440 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- 172.21.15.90:0/3347553011 >> 172.21.15.90:0/3347553011 conn(0x7f1d040b1a50 msgr2=0x7f1d040b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- 172.21.15.90:0/3347553011 shutdown_connections 2024-03-31T15:55:12.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- 172.21.15.90:0/3347553011 wait complete. 2024-03-31T15:55:12.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 Processor -- start 2024-03-31T15:55:12.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- start start 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04072a90 0x7f1d040b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04152150 0x7f1d0414a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d040ca380 con 0x7f1d04151d70 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0415c330 con 0x7f1d04152150 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d0b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d040ca500 con 0x7f1d04072a90 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1cfbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1cfbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34286/0 (socket says 172.21.15.90:34286) 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1cfbe00640 1 -- 172.21.15.90:0/327215003 learned_addr learned my addr 172.21.15.90:0/327215003 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d08a00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04072a90 0x7f1d040b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.069+0000 7f1d09400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04152150 0x7f1d0414a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 -- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04072a90 msgr2=0x7f1d040b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04072a90 0x7f1d040b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 -- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04152150 msgr2=0x7f1d0414a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04152150 0x7f1d0414a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 -- 172.21.15.90:0/327215003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0414ad30 con 0x7f1d04151d70 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1cfbe00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c500 tx=0x7f1cec00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec017070 con 0x7f1d04151d70 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cec005cf0 con 0x7f1d04151d70 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0414b010 con 0x7f1d04151d70 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0072d0 con 0x7f1d04151d70 2024-03-31T15:55:12.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0414b5d0 con 0x7f1d04151d70 2024-03-31T15:55:12.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d04068660 con 0x7f1d04151d70 2024-03-31T15:55:12.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1cec01a070 con 0x7f1d04151d70 2024-03-31T15:55:12.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 0x7f1ce006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f1cec085c10 con 0x7f1d04151d70 2024-03-31T15:55:12.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.073+0000 7f1d08a00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 0x7f1ce006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.077+0000 7f1d08a00640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 0x7f1ce006a840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40ae000 tx=0x7f1cf4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.077+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1cec055c10 con 0x7f1d04151d70 2024-03-31T15:55:12.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.289+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f1d040cdee0 con 0x7f1d04151d70 2024-03-31T15:55:12.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d01600640 1 -- 172.21.15.90:0/327215003 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f1cec059f10 con 0x7f1d04151d70 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 msgr2=0x7f1ce006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 0x7f1ce006a840 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40ae000 tx=0x7f1cf4004740 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 msgr2=0x7f1d0414bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c500 tx=0x7f1cec00c900 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 shutdown_connections 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1ce0068380 0x7f1ce006a840 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d04152150 0x7f1d0414a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d04151d70 0x7f1d0414bdd0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 --2- 172.21.15.90:0/327215003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d04072a90 0x7f1d040b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 >> 172.21.15.90:0/327215003 conn(0x7f1d040b1a50 msgr2=0x7f1d040cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 shutdown_connections 2024-03-31T15:55:12.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.293+0000 7f1d0b400640 1 -- 172.21.15.90:0/327215003 wait complete. 2024-03-31T15:55:12.306 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-31T15:55:12.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 Processor -- start 2024-03-31T15:55:12.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 -- start start 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb74c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb74c0cbd20 con 0x7fb74c06eb20 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb74c0cbfc0 con 0x7fb74c072d30 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb753a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c0cc2a0 con 0x7fb74c15c1d0 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb751a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb74c06eb20 0x7fb74c0b2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43180/0 (socket says 172.21.15.90:43180) 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb751a00640 1 -- 172.21.15.90:0/3572582059 learned_addr learned my addr 172.21.15.90:0/3572582059 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:12.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb74c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 532754672 0 0) 0x7fb74c0cbd20 con 0x7fb74c06eb20 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb730003530 con 0x7fb74c06eb20 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470738070 0 0) 0x7fb74c0cbfc0 con 0x7fb74c072d30 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb74c0cbd20 con 0x7fb74c072d30 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3914700723 0 0) 0x7fb730003530 con 0x7fb74c06eb20 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 msgr2=0x7fb74c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb74c0ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 >> v1:172.21.15.102:6789/0 conn(0x7fb74c072d30 legacy=0x7fb74c06e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c0cc580 con 0x7fb74c06eb20 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fb744001a20 con 0x7fb74c06eb20 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb74c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 >> v1:172.21.15.90:6789/0 conn(0x7fb74c06eb20 legacy=0x7fb74c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb730003b80 0x7fb730003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb730004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730004d50 0x7fb7300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0cc2a0 con 0x7fb730003b80 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb73000bab0 con 0x7fb730004d50 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb73000bcb0 con 0x7fb74c15c1d0 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.449+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb730003b80 0x7fb730003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb74be00640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb730004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751a00640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730004d50 0x7fb7300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 -- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 msgr2=0x7fb730004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb730004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 -- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730004d50 msgr2=0x7fb7300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730004d50 0x7fb7300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 -- 172.21.15.90:0/3572582059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb730003530 con 0x7fb730003b80 2024-03-31T15:55:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb74be00640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15c1d0 0x7fb730004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:12.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751a00640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730004d50 0x7fb7300051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751000640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb730003b80 0x7fb730003f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009cb0 tx=0x7fb73c01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bdd0653885b545 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0b2070 con 0x7fb730003b80 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb73c0a56b0 con 0x7fb730003b80 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb74b400640 1 -- 172.21.15.90:0/3572582059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c009880 con 0x7fb730003b80 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb753a00640 1 -- 172.21.15.90:0/3572582059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb730003b80 msgr2=0x7fb730003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb753a00640 1 --2- 172.21.15.90:0/3572582059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb730003b80 0x7fb730003f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009cb0 tx=0x7fb73c01cc10 comp rx=0 tx=0).stop 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.453+0000 7fb751a00640 1 -- 172.21.15.90:0/3572582059 reap_dead start 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- 172.21.15.90:0/3572582059 shutdown_connections 2024-03-31T15:55:12.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- 172.21.15.90:0/3572582059 >> 172.21.15.90:0/3572582059 conn(0x7fb74c0b1a50 msgr2=0x7fb74c0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- 172.21.15.90:0/3572582059 shutdown_connections 2024-03-31T15:55:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- 172.21.15.90:0/3572582059 wait complete. 2024-03-31T15:55:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 Processor -- start 2024-03-31T15:55:12.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- start start 2024-03-31T15:55:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730003b80 0x7fb74c0c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15edf0 0x7fb74c152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0b3410 con 0x7fb74c15d600 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0b3290 con 0x7fb730003b80 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c0b3590 con 0x7fb74c15edf0 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34300/0 (socket says 172.21.15.90:34300) 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 -- 172.21.15.90:0/3584590558 learned_addr learned my addr 172.21.15.90:0/3584590558 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb74be00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15edf0 0x7fb74c152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751000640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730003b80 0x7fb74c0c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 -- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15edf0 msgr2=0x7fb74c152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15edf0 0x7fb74c152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 -- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730003b80 msgr2=0x7fb74c0c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730003b80 0x7fb74c0c5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 -- 172.21.15.90:0/3584590558 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c0cf1f0 con 0x7fb74c15d600 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb751a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb7440013a0 tx=0x7fb744007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74400ea20 con 0x7fb74c15d600 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb744018070 con 0x7fb74c15d600 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74c0cf4d0 con 0x7fb74c15d600 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.457+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74400ee40 con 0x7fb74c15d600 2024-03-31T15:55:12.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74c0c3be0 con 0x7fb74c15d600 2024-03-31T15:55:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb74c068660 con 0x7fb74c15d600 2024-03-31T15:55:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb74401d460 con 0x7fb74c15d600 2024-03-31T15:55:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb748c00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 0x7fb72806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fb7440869d0 con 0x7fb74c15d600 2024-03-31T15:55:12.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb751000640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 0x7fb72806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.456 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.461+0000 7fb751000640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 0x7fb72806a840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0af000 tx=0x7fb73c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.465+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb744057150 con 0x7fb74c15d600 2024-03-31T15:55:12.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.681+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fb74c153ab0 con 0x7fb74c15d600 2024-03-31T15:55:12.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.681+0000 7fb748c00640 1 -- 172.21.15.90:0/3584590558 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fb744056a50 con 0x7fb74c15d600 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 msgr2=0x7fb72806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 0x7fb72806a840 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0af000 tx=0x7fb73c0046d0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 msgr2=0x7fb74c15e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb7440013a0 tx=0x7fb744007bc0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 shutdown_connections 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb728068380 0x7fb72806a840 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb74c15d600 0x7fb74c15e180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb74c15edf0 0x7fb74c152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 --2- 172.21.15.90:0/3584590558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb730003b80 0x7fb74c0c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 >> 172.21.15.90:0/3584590558 conn(0x7fb74c0b1a50 msgr2=0x7fb74c153fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:12.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 shutdown_connections 2024-03-31T15:55:12.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.685+0000 7fb753a00640 1 -- 172.21.15.90:0/3584590558 wait complete. 2024-03-31T15:55:12.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-31T15:55:12.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.837+0000 7f0f9d400640 1 Processor -- start 2024-03-31T15:55:12.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.837+0000 7f0f9d400640 1 -- start start 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f9d400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f9815f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f9d400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f98071cd0 con 0x7f0f98151d70 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f9d400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f98071fb0 con 0x7f0f98072a90 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f9d400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0f98072290 con 0x7f0f980ccb50 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0f98151d70 0x7f0f980cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43182/0 (socket says 172.21.15.90:43182) 2024-03-31T15:55:12.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 -- 172.21.15.90:0/349432663 learned_addr learned my addr 172.21.15.90:0/349432663 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f97200640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f9815f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925221468 0 0) 0x7f0f98071cd0 con 0x7f0f98151d70 2024-03-31T15:55:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f7c005740 con 0x7f0f98151d70 2024-03-31T15:55:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 744162383 0 0) 0x7f0f7c005740 con 0x7f0f98151d70 2024-03-31T15:55:12.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 msgr2=0x7f0f9815f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f9815f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 >> v1:172.21.15.102:6789/0 conn(0x7f0f98072a90 legacy=0x7f0f9815efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f98072570 con 0x7f0f98151d70 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f0f88001a00 con 0x7f0f98151d70 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 >> v1:172.21.15.90:6789/0 conn(0x7f0f98151d70 legacy=0x7f0f980cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7c005c00 0x7f0f7c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f7c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c006f60 0x7f0f7c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f98072290 con 0x7f0f7c005c00 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c00dbe0 con 0x7f0f7c006f60 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0f7c00dde0 con 0x7f0f980ccb50 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7c005c00 0x7f0f7c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95e00640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f7c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f97200640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c006f60 0x7f0f7c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 -- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 msgr2=0x7f0f7c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f7c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 -- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c006f60 msgr2=0x7f0f7c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c006f60 0x7f0f7c0073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f96800640 1 -- 172.21.15.90:0/349432663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c005740 con 0x7f0f7c005c00 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f97200640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c006f60 0x7f0f7c0073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.841+0000 7f0f95e00640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f980ccb50 0x7f0f7c006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:12.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f96800640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7c005c00 0x7f0f7c005fe0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800a920 tx=0x7f0f8801d380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30835e763f50a6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8800a4c0 con 0x7f0f7c005c00 2024-03-31T15:55:12.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f880ad650 con 0x7f0f7c005c00 2024-03-31T15:55:12.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f95400640 1 -- 172.21.15.90:0/349432663 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f880ada40 con 0x7f0f7c005c00 2024-03-31T15:55:12.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- 172.21.15.90:0/349432663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7c005c00 msgr2=0x7f0f7c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 --2- 172.21.15.90:0/349432663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f7c005c00 0x7f0f7c005fe0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800a920 tx=0x7f0f8801d380 comp rx=0 tx=0).stop 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f97200640 1 -- 172.21.15.90:0/349432663 reap_dead start 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- 172.21.15.90:0/349432663 shutdown_connections 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- 172.21.15.90:0/349432663 >> 172.21.15.90:0/349432663 conn(0x7f0f980b1a50 msgr2=0x7f0f980b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- 172.21.15.90:0/349432663 shutdown_connections 2024-03-31T15:55:12.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- 172.21.15.90:0/349432663 wait complete. 2024-03-31T15:55:12.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 Processor -- start 2024-03-31T15:55:12.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.845+0000 7f0f9d400640 1 -- start start 2024-03-31T15:55:12.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c005c00 0x7f0f9814f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f98158330 0x7f0f981587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9806cfa0 con 0x7f0f980c8e10 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f9815c370 con 0x7f0f7c005c00 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0f9806d120 con 0x7f0f98158330 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56480/0 (socket says 172.21.15.90:56480) 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 -- 172.21.15.90:0/334960527 learned_addr learned my addr 172.21.15.90:0/334960527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f95e00640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f98158330 0x7f0f981587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f96800640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c005c00 0x7f0f9814f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 -- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f98158330 msgr2=0x7f0f981587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f98158330 0x7f0f981587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 -- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c005c00 msgr2=0x7f0f9814f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:12.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c005c00 0x7f0f9814f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:12.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 -- 172.21.15.90:0/334960527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f980b2e00 con 0x7f0f980c8e10 2024-03-31T15:55:12.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f97200640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0f980733e0 tx=0x7f0f90011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9001d070 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f90004250 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f980b30e0 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f90015730 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f980b36a0 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f98068660 con 0x7f0f980c8e10 2024-03-31T15:55:12.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.849+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0f90020070 con 0x7f0f980c8e10 2024-03-31T15:55:12.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.853+0000 7f0f8ea00640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 0x7f0f7806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:12.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.853+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f0f90085ec0 con 0x7f0f980c8e10 2024-03-31T15:55:12.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.853+0000 7f0f96800640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 0x7f0f7806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:12.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.853+0000 7f0f96800640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 0x7f0f7806a870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f880ad000 tx=0x7f0f880a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:12.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:12.857+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0f90055f50 con 0x7f0f980c8e10 2024-03-31T15:55:13.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.069+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f0f980cdee0 con 0x7f0f980c8e10 2024-03-31T15:55:13.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f8ea00640 1 -- 172.21.15.90:0/334960527 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f0f9005a250 con 0x7f0f980c8e10 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 msgr2=0x7f0f7806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 0x7f0f7806a870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f880ad000 tx=0x7f0f880a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 msgr2=0x7f0f980c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0f980733e0 tx=0x7f0f90011920 comp rx=0 tx=0).stop 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 shutdown_connections 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0f780683b0 0x7f0f7806a870 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f980c8e10 0x7f0f980c9290 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0f98158330 0x7f0f981587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 --2- 172.21.15.90:0/334960527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f7c005c00 0x7f0f9814f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 >> 172.21.15.90:0/334960527 conn(0x7f0f980b1a50 msgr2=0x7f0f980cfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 shutdown_connections 2024-03-31T15:55:13.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.073+0000 7f0f9d400640 1 -- 172.21.15.90:0/334960527 wait complete. 2024-03-31T15:55:13.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-31T15:55:13.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 Processor -- start 2024-03-31T15:55:13.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 -- start start 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5ac15aaf0 0x7fd5ac0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0cf3c0 0x7fd5ac15c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5ac06b790 con 0x7fd5ac0b3550 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd5ac0b4320 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5b1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ac15bee0 con 0x7fd5ac0cf3c0 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5aa800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd5ac15aaf0 0x7fd5ac0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5ab200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5ac0b3550 0x7fd5ac06c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52248/0 (socket says 172.21.15.90:52248) 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5ab200640 1 -- 172.21.15.90:0/1798482241 learned_addr learned my addr 172.21.15.90:0/1798482241 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.229+0000 7fd5a1e00640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0cf3c0 0x7fd5ac15c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5aa800640 1 -- 172.21.15.90:0/1798482241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0cf3c0 msgr2=0x7fd5ac15c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5aa800640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0cf3c0 0x7fd5ac15c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5aa800640 1 -- 172.21.15.90:0/1798482241 >> v1:172.21.15.90:6789/0 conn(0x7fd5ac0b3550 legacy=0x7fd5ac06c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5aa800640 1 -- 172.21.15.90:0/1798482241 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ac06ba70 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5a9e00640 1 -- 172.21.15.90:0/1798482241 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975623602 0 0) 0x7fd5ac06b790 con 0x7fd5ac0b3550 2024-03-31T15:55:13.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5aa800640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0afb60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00a960 tx=0x7fd59c01d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be45961f9bf2959d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5a9e00640 1 -- 172.21.15.90:0/1798482241 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0b3070 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5a9e00640 1 -- 172.21.15.90:0/1798482241 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59c0a6850 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.233+0000 7fd5a9e00640 1 -- 172.21.15.90:0/1798482241 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00b040 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- 172.21.15.90:0/1798482241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 msgr2=0x7fd5ac0afb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0afb60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00a960 tx=0x7fd59c01d870 comp rx=0 tx=0).stop 2024-03-31T15:55:13.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- 172.21.15.90:0/1798482241 shutdown_connections 2024-03-31T15:55:13.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0cf3c0 0x7fd5ac15c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- 172.21.15.90:0/1798482241 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0afb60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- 172.21.15.90:0/1798482241 >> 172.21.15.90:0/1798482241 conn(0x7fd5ac0cd060 msgr2=0x7fd5ac0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- 172.21.15.90:0/1798482241 shutdown_connections 2024-03-31T15:55:13.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- 172.21.15.90:0/1798482241 wait complete. 2024-03-31T15:55:13.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 Processor -- start 2024-03-31T15:55:13.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- start start 2024-03-31T15:55:13.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac0cf3c0 0x7fd5ac14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ac15c840 con 0x7fd5ac15aaf0 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ac15c6c0 con 0x7fd5ac0cf3c0 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5b1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd5ac167600 con 0x7fd5ac0b3550 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5ab200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5a1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac0cf3c0 0x7fd5ac14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5ab200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56502/0 (socket says 172.21.15.90:56502) 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:45622/0 (socket says 172.21.15.90:45622) 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5ab200640 1 -- 172.21.15.90:0/3520194857 learned_addr learned my addr 172.21.15.90:0/3520194857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 -- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac0cf3c0 msgr2=0x7fd5ac14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac0cf3c0 0x7fd5ac14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 -- 172.21.15.90:0/3520194857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 msgr2=0x7fd5ac0b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 -- 172.21.15.90:0/3520194857 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ac152810 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.237+0000 7fd5aa800640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0ad000 tx=0x7fd59c01a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0b3040 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59c0bd070 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0ae880 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ac152af0 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ac157270 con 0x7fd5ac0b3550 2024-03-31T15:55:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ac068660 con 0x7fd5ac0b3550 2024-03-31T15:55:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd59c0a6710 con 0x7fd5ac0b3550 2024-03-31T15:55:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 0x7fd58806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fd59c120af0 con 0x7fd5ac0b3550 2024-03-31T15:55:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.241+0000 7fd5a1e00640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 0x7fd58806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.245+0000 7fd5a1e00640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 0x7fd58806a840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac157ff0 tx=0x7fd594009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.245+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd59c017460 con 0x7fd5ac0b3550 2024-03-31T15:55:13.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.457+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fd5ac152cc0 con 0x7fd5ac0b3550 2024-03-31T15:55:13.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5a3400640 1 -- 172.21.15.90:0/3520194857 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fd59c0f0c00 con 0x7fd5ac0b3550 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 msgr2=0x7fd58806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 0x7fd58806a840 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac157ff0 tx=0x7fd594009190 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 msgr2=0x7fd5ac157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0ad000 tx=0x7fd59c01a610 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 shutdown_connections 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd588068380 0x7fd58806a840 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5ac15aaf0 0x7fd5ac0b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac0cf3c0 0x7fd5ac14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 --2- 172.21.15.90:0/3520194857 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd5ac0b3550 0x7fd5ac157790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 >> 172.21.15.90:0/3520194857 conn(0x7fd5ac0cd060 msgr2=0x7fd5ac1594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:13.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 shutdown_connections 2024-03-31T15:55:13.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.465+0000 7fd5b1400640 1 -- 172.21.15.90:0/3520194857 wait complete. 2024-03-31T15:55:13.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-31T15:55:13.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.621+0000 7ff575400640 1 Processor -- start 2024-03-31T15:55:13.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.621+0000 7ff575400640 1 -- start start 2024-03-31T15:55:13.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff575400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff5700ccb50 0x7ff57015f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff575400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff570071cd0 con 0x7ff570151d70 2024-03-31T15:55:13.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff575400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff570071fb0 con 0x7ff570072a90 2024-03-31T15:55:13.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff575400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff570072290 con 0x7ff5700ccb50 2024-03-31T15:55:13.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff5700ccb50 0x7ff57015f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff570151d70 0x7ff5700cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52260/0 (socket says 172.21.15.90:52260) 2024-03-31T15:55:13.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56e800640 1 -- 172.21.15.90:0/3254503061 learned_addr learned my addr 172.21.15.90:0/3254503061 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:13.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56f200640 1 -- 172.21.15.90:0/3254503061 >> v1:172.21.15.102:6789/0 conn(0x7ff570072a90 legacy=0x7ff57015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56f200640 1 -- 172.21.15.90:0/3254503061 >> v1:172.21.15.90:6789/0 conn(0x7ff570151d70 legacy=0x7ff5700cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.625+0000 7ff56f200640 1 -- 172.21.15.90:0/3254503061 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff570072570 con 0x7ff5700ccb50 2024-03-31T15:55:13.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff56f200640 1 --2- 172.21.15.90:0/3254503061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5700ccb50 0x7ff57015f6f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff568009d90 tx=0x7ff56801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8ee187f4b71d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff56de00640 1 -- 172.21.15.90:0/3254503061 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680b2070 con 0x7ff5700ccb50 2024-03-31T15:55:13.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff56de00640 1 -- 172.21.15.90:0/3254503061 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5680a5a50 con 0x7ff5700ccb50 2024-03-31T15:55:13.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff56de00640 1 -- 172.21.15.90:0/3254503061 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680a5e20 con 0x7ff5700ccb50 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- 172.21.15.90:0/3254503061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5700ccb50 msgr2=0x7ff57015f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 --2- 172.21.15.90:0/3254503061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5700ccb50 0x7ff57015f6f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff568009d90 tx=0x7ff56801c970 comp rx=0 tx=0).stop 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- 172.21.15.90:0/3254503061 shutdown_connections 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 --2- 172.21.15.90:0/3254503061 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff5700ccb50 0x7ff57015f6f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- 172.21.15.90:0/3254503061 >> 172.21.15.90:0/3254503061 conn(0x7ff5700b1a50 msgr2=0x7ff5700b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- 172.21.15.90:0/3254503061 shutdown_connections 2024-03-31T15:55:13.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- 172.21.15.90:0/3254503061 wait complete. 2024-03-31T15:55:13.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 Processor -- start 2024-03-31T15:55:13.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.629+0000 7ff575400640 1 -- start start 2024-03-31T15:55:13.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff570072a90 0x7ff5701526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff5700ccb50 0x7ff57014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff570168ec0 con 0x7ff570151d70 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff57015c330 con 0x7ff570072a90 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff570169040 con 0x7ff5700ccb50 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56518/0 (socket says 172.21.15.90:56518) 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 -- 172.21.15.90:0/1508370233 learned_addr learned my addr 172.21.15.90:0/1508370233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff565e00640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff5700ccb50 0x7ff57014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56e800640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff570072a90 0x7ff5701526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 -- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff5700ccb50 msgr2=0x7ff57014c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff5700ccb50 0x7ff57014c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 -- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff570072a90 msgr2=0x7ff5701526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff570072a90 0x7ff5701526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 -- 172.21.15.90:0/1508370233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff570149260 con 0x7ff570151d70 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff56f200640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff56801a5a0 tx=0x7ff568019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680b2050 con 0x7ff570151d70 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5680be070 con 0x7ff570151d70 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5680b7830 con 0x7ff570151d70 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff570149540 con 0x7ff570151d70 2024-03-31T15:55:13.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff570148c80 con 0x7ff570151d70 2024-03-31T15:55:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff570068660 con 0x7ff570151d70 2024-03-31T15:55:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.633+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5680b79d0 con 0x7ff570151d70 2024-03-31T15:55:13.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.637+0000 7ff567400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 0x7ff54c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.637+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff568122090 con 0x7ff570151d70 2024-03-31T15:55:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.637+0000 7ff56e800640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 0x7ff54c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:13.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.637+0000 7ff56e800640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 0x7ff54c06a840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff560005890 tx=0x7ff560009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:13.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.641+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff568016560 con 0x7ff570151d70 2024-03-31T15:55:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.853+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7ff5700c7c30 con 0x7ff570151d70 2024-03-31T15:55:13.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.853+0000 7ff567400640 1 -- 172.21.15.90:0/1508370233 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7ff5680f2430 con 0x7ff570151d70 2024-03-31T15:55:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 msgr2=0x7ff54c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 0x7ff54c06a840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff560005890 tx=0x7ff560009040 comp rx=0 tx=0).stop 2024-03-31T15:55:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 msgr2=0x7ff570148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:13.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff56801a5a0 tx=0x7ff568019710 comp rx=0 tx=0).stop 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 shutdown_connections 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff54c068380 0x7ff54c06a840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff570151d70 0x7ff570148740 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff5700ccb50 0x7ff57014c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 --2- 172.21.15.90:0/1508370233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff570072a90 0x7ff5701526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 >> 172.21.15.90:0/1508370233 conn(0x7ff5700b1a50 msgr2=0x7ff5700ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 shutdown_connections 2024-03-31T15:55:13.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:13.857+0000 7ff575400640 1 -- 172.21.15.90:0/1508370233 wait complete. 2024-03-31T15:55:13.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-31T15:55:14.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 Processor -- start 2024-03-31T15:55:14.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 -- start start 2024-03-31T15:55:14.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b0015ec20 0x7f6b000b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b000b5250 con 0x7f6b000addf0 2024-03-31T15:55:14.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6b00158e90 con 0x7f6b0015ec20 2024-03-31T15:55:14.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6b06400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b000b5020 con 0x7f6b000af890 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:45640/0 (socket says 172.21.15.90:45640) 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6aff400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b0015ec20 0x7f6b000b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 -- 172.21.15.90:0/63200901 learned_addr learned my addr 172.21.15.90:0/63200901 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 -- 172.21.15.90:0/63200901 >> v2:172.21.15.102:3300/0 conn(0x7f6b0015ec20 msgr2=0x7f6b000b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 --2- 172.21.15.90:0/63200901 >> v2:172.21.15.102:3300/0 conn(0x7f6b0015ec20 0x7f6b000b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 -- 172.21.15.90:0/63200901 >> v1:172.21.15.102:6789/0 conn(0x7f6b000addf0 legacy=0x7f6b0015f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.013+0000 7f6afea00640 1 -- 172.21.15.90:0/63200901 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b000b5530 con 0x7f6b000af890 2024-03-31T15:55:14.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6afea00640 1 --2- 172.21.15.90:0/63200901 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009870 tx=0x7f6aec01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771136fbba1e9b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6afe000640 1 -- 172.21.15.90:0/63200901 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0ab650 con 0x7f6b000af890 2024-03-31T15:55:14.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6afe000640 1 -- 172.21.15.90:0/63200901 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aec0b0070 con 0x7f6b000af890 2024-03-31T15:55:14.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6afe000640 1 -- 172.21.15.90:0/63200901 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0aba10 con 0x7f6b000af890 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 -- 172.21.15.90:0/63200901 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 msgr2=0x7f6b000ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 --2- 172.21.15.90:0/63200901 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009870 tx=0x7f6aec01c560 comp rx=0 tx=0).stop 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 -- 172.21.15.90:0/63200901 shutdown_connections 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 --2- 172.21.15.90:0/63200901 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000af890 0x7f6b000ad8b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 --2- 172.21.15.90:0/63200901 >> v2:172.21.15.102:3300/0 conn(0x7f6b0015ec20 0x7f6b000b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 -- 172.21.15.90:0/63200901 >> 172.21.15.90:0/63200901 conn(0x7f6b000cd060 msgr2=0x7f6b000cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 -- 172.21.15.90:0/63200901 shutdown_connections 2024-03-31T15:55:14.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 -- 172.21.15.90:0/63200901 wait complete. 2024-03-31T15:55:14.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.017+0000 7f6b06400640 1 Processor -- start 2024-03-31T15:55:14.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- start start 2024-03-31T15:55:14.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000addf0 0x7f6b00071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b0015ec20 0x7f6b00156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0016af90 con 0x7f6b000af890 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0015f710 con 0x7f6b0015ec20 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0016b110 con 0x7f6b000addf0 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56534/0 (socket says 172.21.15.90:56534) 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 -- 172.21.15.90:0/2974468406 learned_addr learned my addr 172.21.15.90:0/2974468406 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6aff400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000addf0 0x7f6b00071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6affe00640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b0015ec20 0x7f6b00156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 -- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000addf0 msgr2=0x7f6b00071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000addf0 0x7f6b00071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 -- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b0015ec20 msgr2=0x7f6b00156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b0015ec20 0x7f6b00156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 -- 172.21.15.90:0/2974468406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b00072180 con 0x7f6b000af890 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6afea00640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0ad040 tx=0x7f6aec01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0be5f0 con 0x7f6b000af890 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aec0be790 con 0x7f6b000af890 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0beb80 con 0x7f6b000af890 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b000b02f0 con 0x7f6b000af890 2024-03-31T15:55:14.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b000b08b0 con 0x7f6b000af890 2024-03-31T15:55:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.021+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b00068660 con 0x7f6b000af890 2024-03-31T15:55:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.025+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6aec0b43f0 con 0x7f6b000af890 2024-03-31T15:55:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.025+0000 7f6af7400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 0x7f6ae006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.025+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6aec1212b0 con 0x7f6b000af890 2024-03-31T15:55:14.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.025+0000 7f6aff400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 0x7f6ae006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.025+0000 7f6aff400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 0x7f6ae006a840 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6b001fa9b0 tx=0x7f6af0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.029+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6aec016150 con 0x7f6b000af890 2024-03-31T15:55:14.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.240+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f6b001529d0 con 0x7f6b000af890 2024-03-31T15:55:14.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.240+0000 7f6af7400640 1 -- 172.21.15.90:0/2974468406 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7f6aec1210d0 con 0x7f6b000af890 2024-03-31T15:55:14.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 msgr2=0x7f6ae006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 0x7f6ae006a840 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6b001fa9b0 tx=0x7f6af0009190 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 msgr2=0x7f6b0014cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0ad040 tx=0x7f6aec01ada0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 shutdown_connections 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ae0068380 0x7f6ae006a840 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b0015ec20 0x7f6b00156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b000af890 0x7f6b0014cc70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 --2- 172.21.15.90:0/2974468406 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6b000addf0 0x7f6b00071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 >> 172.21.15.90:0/2974468406 conn(0x7f6b000cd060 msgr2=0x7f6b000ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 shutdown_connections 2024-03-31T15:55:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.244+0000 7f6b06400640 1 -- 172.21.15.90:0/2974468406 wait complete. 2024-03-31T15:55:14.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:14.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.396+0000 7f6f4b200640 1 Processor -- start 2024-03-31T15:55:14.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.396+0000 7f6f4b200640 1 -- start start 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f4b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 0x7f6f4406b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f4b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f4406bd60 0x7f6f4415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f4b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f440b47e0 con 0x7f6f441449a0 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f4b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6f440b4ac0 con 0x7f6f4406bd60 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f4b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6f4415b7d0 con 0x7f6f4415bfd0 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6f441449a0 0x7f6f440b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52268/0 (socket says 172.21.15.90:52268) 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43e00640 1 -- 172.21.15.90:0/1508448698 learned_addr learned my addr 172.21.15.90:0/1508448698 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43400640 1 --2- 172.21.15.90:0/1508448698 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 0x7f6f4406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f49200640 1 --2- 172.21.15.90:0/1508448698 >> v2:172.21.15.102:3300/0 conn(0x7f6f4406bd60 0x7f6f4415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f42a00640 1 -- 172.21.15.90:0/1508448698 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116920907 0 0) 0x7f6f440b47e0 con 0x7f6f441449a0 2024-03-31T15:55:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f42a00640 1 -- 172.21.15.90:0/1508448698 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f28003530 con 0x7f6f441449a0 2024-03-31T15:55:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43400640 1 -- 172.21.15.90:0/1508448698 >> v2:172.21.15.102:3300/0 conn(0x7f6f4406bd60 msgr2=0x7f6f4415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43400640 1 --2- 172.21.15.90:0/1508448698 >> v2:172.21.15.102:3300/0 conn(0x7f6f4406bd60 0x7f6f4415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43400640 1 -- 172.21.15.90:0/1508448698 >> v1:172.21.15.90:6789/0 conn(0x7f6f441449a0 legacy=0x7f6f440b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.400+0000 7f6f43400640 1 -- 172.21.15.90:0/1508448698 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4415bab0 con 0x7f6f4415bfd0 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f43400640 1 --2- 172.21.15.90:0/1508448698 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 0x7f6f4406b820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0097c0 tx=0x7f6f2c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e501f24f319234e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f42a00640 1 -- 172.21.15.90:0/1508448698 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0b0070 con 0x7f6f4415bfd0 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f42a00640 1 -- 172.21.15.90:0/1508448698 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0ab600 con 0x7f6f4415bfd0 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f42a00640 1 -- 172.21.15.90:0/1508448698 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0ab9b0 con 0x7f6f4415bfd0 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- 172.21.15.90:0/1508448698 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 msgr2=0x7f6f4406b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 --2- 172.21.15.90:0/1508448698 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 0x7f6f4406b820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0097c0 tx=0x7f6f2c01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- 172.21.15.90:0/1508448698 shutdown_connections 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 --2- 172.21.15.90:0/1508448698 >> v2:172.21.15.102:3300/0 conn(0x7f6f4406bd60 0x7f6f4415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 --2- 172.21.15.90:0/1508448698 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f4415bfd0 0x7f6f4406b820 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- 172.21.15.90:0/1508448698 >> 172.21.15.90:0/1508448698 conn(0x7f6f440b1c80 msgr2=0x7f6f440b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- 172.21.15.90:0/1508448698 shutdown_connections 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- 172.21.15.90:0/1508448698 wait complete. 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 Processor -- start 2024-03-31T15:55:14.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.404+0000 7f6f4b200640 1 -- start start 2024-03-31T15:55:14.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 0x7f6f44143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 0x7f6f44156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6f44167f70 con 0x7f6f4406bd60 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f44167df0 con 0x7f6f4415bfd0 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6f441680f0 con 0x7f6f441449a0 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56542/0 (socket says 172.21.15.90:56542) 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 -- 172.21.15.90:0/4151808394 learned_addr learned my addr 172.21.15.90:0/4151808394 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f49200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 0x7f6f44156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43400640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 0x7f6f44143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 -- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 msgr2=0x7f6f44143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 0x7f6f44143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 -- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 msgr2=0x7f6f44156880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 0x7f6f44156880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 -- 172.21.15.90:0/4151808394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f441f7000 con 0x7f6f4406bd60 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f49200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 0x7f6f44156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43400640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 0x7f6f44143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f43e00640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c3c0 tx=0x7f6f3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34017070 con 0x7f6f4406bd60 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f34005cf0 con 0x7f6f4406bd60 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f340072f0 con 0x7f6f4406bd60 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f441f72e0 con 0x7f6f4406bd60 2024-03-31T15:55:14.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f441f78a0 con 0x7f6f4406bd60 2024-03-31T15:55:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f440686d0 con 0x7f6f4406bd60 2024-03-31T15:55:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.408+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6f3401a070 con 0x7f6f4406bd60 2024-03-31T15:55:14.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.412+0000 7f6f3be00640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 0x7f6f2006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.412+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6f34085cb0 con 0x7f6f4406bd60 2024-03-31T15:55:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.412+0000 7f6f43400640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 0x7f6f2006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.412+0000 7f6f43400640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 0x7f6f2006a840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0ad000 tx=0x7f6f2c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.416+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6f34056390 con 0x7f6f4406bd60 2024-03-31T15:55:14.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.572+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f44072050 con 0x7f6f4406bd60 2024-03-31T15:55:14.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.572+0000 7f6f3be00640 1 -- 172.21.15.90:0/4151808394 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13128 (secure 0 0 0) 0x7f6f34055c90 con 0x7f6f4406bd60 2024-03-31T15:55:14.568 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:14.568 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":35,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:11.520087+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 msgr2=0x7f6f2006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 0x7f6f2006a840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0ad000 tx=0x7f6f2c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 msgr2=0x7f6f44142bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c3c0 tx=0x7f6f3400c880 comp rx=0 tx=0).stop 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 shutdown_connections 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6f20068380 0x7f6f2006a840 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f4415bfd0 0x7f6f44156880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6f441449a0 0x7f6f44143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 --2- 172.21.15.90:0/4151808394 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f4406bd60 0x7f6f44142bf0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 >> 172.21.15.90:0/4151808394 conn(0x7f6f440b1c80 msgr2=0x7f6f4413ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 shutdown_connections 2024-03-31T15:55:14.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.576+0000 7f6f4b200640 1 -- 172.21.15.90:0/4151808394 wait complete. 2024-03-31T15:55:14.588 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:14.588 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:14.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 Processor -- start 2024-03-31T15:55:14.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 -- start start 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 0x7fd6c806fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c815f6a0 con 0x7fd6c80b3cc0 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6c80b4230 con 0x7fd6c806b5f0 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6d0200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c80b4510 con 0x7fd6c815f210 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6cce00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd6c80b3cc0 0x7fd6c806f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52276/0 (socket says 172.21.15.90:52276) 2024-03-31T15:55:14.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.732+0000 7fd6cce00640 1 -- 172.21.15.90:0/2561483205 learned_addr learned my addr 172.21.15.90:0/2561483205 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6ce200640 1 --2- 172.21.15.90:0/2561483205 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 0x7fd6c806fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167594347 0 0) 0x7fd6c815f6a0 con 0x7fd6c80b3cc0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6b0003530 con 0x7fd6c80b3cc0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946709817 0 0) 0x7fd6c80b4230 con 0x7fd6c806b5f0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6c815f6a0 con 0x7fd6c806b5f0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1177814202 0 0) 0x7fd6b0003530 con 0x7fd6c80b3cc0 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6ce200640 1 -- 172.21.15.90:0/2561483205 >> v1:172.21.15.102:6789/0 conn(0x7fd6c806b5f0 legacy=0x7fd6c806b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6ce200640 1 -- 172.21.15.90:0/2561483205 >> v1:172.21.15.90:6789/0 conn(0x7fd6c80b3cc0 legacy=0x7fd6c806f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6ce200640 1 -- 172.21.15.90:0/2561483205 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c8161fb0 con 0x7fd6c815f210 2024-03-31T15:55:14.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.736+0000 7fd6ce200640 1 --2- 172.21.15.90:0/2561483205 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 0x7fd6c806fa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009dc0 tx=0x7fd6c001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f81159e016056b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4001680 con 0x7fd6c815f210 2024-03-31T15:55:14.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c00a6500 con 0x7fd6c815f210 2024-03-31T15:55:14.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6c7e00640 1 -- 172.21.15.90:0/2561483205 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00a68b0 con 0x7fd6c815f210 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- 172.21.15.90:0/2561483205 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 msgr2=0x7fd6c806fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2561483205 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 0x7fd6c806fa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009dc0 tx=0x7fd6c001c8b0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- 172.21.15.90:0/2561483205 shutdown_connections 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2561483205 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c815f210 0x7fd6c806fa20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- 172.21.15.90:0/2561483205 >> 172.21.15.90:0/2561483205 conn(0x7fd6c80b29c0 msgr2=0x7fd6c80b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- 172.21.15.90:0/2561483205 shutdown_connections 2024-03-31T15:55:14.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- 172.21.15.90:0/2561483205 wait complete. 2024-03-31T15:55:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 Processor -- start 2024-03-31T15:55:14.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- start start 2024-03-31T15:55:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c806b5f0 0x7fd6c8147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6c80b3cc0 0x7fd6c814d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c816c920 con 0x7fd6c815f210 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80b5d20 con 0x7fd6c80b3cc0 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6d0200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c816caa0 con 0x7fd6c806b5f0 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6ce200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6ce200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56552/0 (socket says 172.21.15.90:56552) 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6ce200640 1 -- 172.21.15.90:0/2409378064 learned_addr learned my addr 172.21.15.90:0/2409378064 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6cce00640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6c80b3cc0 0x7fd6c814d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.740+0000 7fd6cd800640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c806b5f0 0x7fd6c8147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 -- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c806b5f0 msgr2=0x7fd6c8147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c806b5f0 0x7fd6c8147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 -- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6c80b3cc0 msgr2=0x7fd6c814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6c80b3cc0 0x7fd6c814d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 -- 172.21.15.90:0/2409378064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c814aa70 con 0x7fd6c815f210 2024-03-31T15:55:14.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6ce200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd6c001cd90 tx=0x7fd6c0019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0009450 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c00c1070 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c814ad50 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b8aa0 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c814b310 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c8075880 con 0x7fd6c815f210 2024-03-31T15:55:14.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6c00b8c40 con 0x7fd6c815f210 2024-03-31T15:55:14.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 0x7fd6a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:14.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fd6c0121780 con 0x7fd6c815f210 2024-03-31T15:55:14.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.744+0000 7fd6cd800640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 0x7fd6a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:14.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.748+0000 7fd6cd800640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 0x7fd6a806a840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0011f0 tx=0x7fd6bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:14.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.748+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd6c00164a0 con 0x7fd6c815f210 2024-03-31T15:55:14.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.904+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6c80685f0 con 0x7fd6c815f210 2024-03-31T15:55:14.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.904+0000 7fd6c5600640 1 -- 172.21.15.90:0/2409378064 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fd6c00f18a0 con 0x7fd6c815f210 2024-03-31T15:55:14.899 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 msgr2=0x7fd6a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 0x7fd6a806a840 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0011f0 tx=0x7fd6bc009040 comp rx=0 tx=0).stop 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 msgr2=0x7fd6c814e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd6c001cd90 tx=0x7fd6c0019650 comp rx=0 tx=0).stop 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 shutdown_connections 2024-03-31T15:55:14.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd6a8068380 0x7fd6a806a840 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6c815f210 0x7fd6c814e540 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6c80b3cc0 0x7fd6c814d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 --2- 172.21.15.90:0/2409378064 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6c806b5f0 0x7fd6c8147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 >> 172.21.15.90:0/2409378064 conn(0x7fd6c80b29c0 msgr2=0x7fd6c8152550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 shutdown_connections 2024-03-31T15:55:14.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:14.908+0000 7fd6d0200640 1 -- 172.21.15.90:0/2409378064 wait complete. 2024-03-31T15:55:14.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:55:15.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.060+0000 7f279ca00640 1 Processor -- start 2024-03-31T15:55:15.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.060+0000 7f279ca00640 1 -- start start 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f27981552d0 0x7f279806b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2798070260 0x7f27980b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798161d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f27980b41f0 con 0x7f27981552d0 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f27981556b0 con 0x7f2798070260 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f279ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f279806bb30 con 0x7f27980b4bc0 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f27981552d0 0x7f279806b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f27981552d0 0x7f279806b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56556/0 (socket says 172.21.15.90:56556) 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 -- 172.21.15.90:0/1209458060 learned_addr learned my addr 172.21.15.90:0/1209458060 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2796800640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798161d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795400640 1 --2- 172.21.15.90:0/1209458060 >> v2:172.21.15.102:3300/0 conn(0x7f2798070260 0x7f27980b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 -- 172.21.15.90:0/1209458060 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 msgr2=0x7f2798161d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798161d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 -- 172.21.15.90:0/1209458060 >> v2:172.21.15.102:3300/0 conn(0x7f2798070260 msgr2=0x7f27980b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 --2- 172.21.15.90:0/1209458060 >> v2:172.21.15.102:3300/0 conn(0x7f2798070260 0x7f27980b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.064+0000 7f2795e00640 1 -- 172.21.15.90:0/1209458060 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27980ae4a0 con 0x7f27981552d0 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f2795e00640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f279806b5f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f27980b3700 tx=0x7f278001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9e4ee47e221988 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f2794a00640 1 -- 172.21.15.90:0/1209458060 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800b1070 con 0x7f27981552d0 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f2794a00640 1 -- 172.21.15.90:0/1209458060 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27800ac630 con 0x7f27981552d0 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f2794a00640 1 -- 172.21.15.90:0/1209458060 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27800a8050 con 0x7f27981552d0 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- 172.21.15.90:0/1209458060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 msgr2=0x7f279806b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f279806b5f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f27980b3700 tx=0x7f278001c520 comp rx=0 tx=0).stop 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- 172.21.15.90:0/1209458060 shutdown_connections 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798161d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 --2- 172.21.15.90:0/1209458060 >> v2:172.21.15.102:3300/0 conn(0x7f2798070260 0x7f27980b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 --2- 172.21.15.90:0/1209458060 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f279806b5f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- 172.21.15.90:0/1209458060 >> 172.21.15.90:0/1209458060 conn(0x7f27980d0650 msgr2=0x7f27980d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:15.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- 172.21.15.90:0/1209458060 shutdown_connections 2024-03-31T15:55:15.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- 172.21.15.90:0/1209458060 wait complete. 2024-03-31T15:55:15.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 Processor -- start 2024-03-31T15:55:15.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.068+0000 7f279ca00640 1 -- start start 2024-03-31T15:55:15.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798070260 0x7f279814c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2798162ab0 con 0x7f27981552d0 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f27980aea60 con 0x7f2798070260 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f27980b4fd0 con 0x7f27980b4bc0 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56560/0 (socket says 172.21.15.90:56560) 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 -- 172.21.15.90:0/1280907825 learned_addr learned my addr 172.21.15.90:0/1280907825 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2795e00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798070260 0x7f279814c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2795400640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 -- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 msgr2=0x7f2798158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 -- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798070260 msgr2=0x7f279814c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798070260 0x7f279814c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 -- 172.21.15.90:0/1280907825 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798156220 con 0x7f27981552d0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2795400640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f2796800640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2790007e70 tx=0x7f279000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790014620 con 0x7f27981552d0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2790019070 con 0x7f27981552d0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790014a20 con 0x7f27981552d0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2798159580 con 0x7f27981552d0 2024-03-31T15:55:15.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981564c0 con 0x7f27981552d0 2024-03-31T15:55:15.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27980686d0 con 0x7f27981552d0 2024-03-31T15:55:15.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.072+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2790014bc0 con 0x7f27981552d0 2024-03-31T15:55:15.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.076+0000 7f278e000640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 0x7f277406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.076+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f2790087120 con 0x7f27981552d0 2024-03-31T15:55:15.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.076+0000 7f2795e00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 0x7f277406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.076+0000 7f2795e00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 0x7f277406a840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27800ae000 tx=0x7f2780008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.080+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2790011260 con 0x7f27981552d0 2024-03-31T15:55:15.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.236+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f27980b3bc0 con 0x7f27981552d0 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.236+0000 7f278e000640 1 -- 172.21.15.90:0/1280907825 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f2790057890 con 0x7f27981552d0 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:15.232 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 msgr2=0x7f277406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 0x7f277406a840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f27800ae000 tx=0x7f2780008040 comp rx=0 tx=0).stop 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 msgr2=0x7f2798157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2790007e70 tx=0x7f279000e4b0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 shutdown_connections 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2774068380 0x7f277406a840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f27981552d0 0x7f2798157f40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f27980b4bc0 0x7f2798158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 --2- 172.21.15.90:0/1280907825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798070260 0x7f279814c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 >> 172.21.15.90:0/1280907825 conn(0x7f27980d0650 msgr2=0x7f279815dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 shutdown_connections 2024-03-31T15:55:15.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.240+0000 7f279ca00640 1 -- 172.21.15.90:0/1280907825 wait complete. 2024-03-31T15:55:15.252 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:55:15.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:55:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.396+0000 7f9cd7c00640 1 Processor -- start 2024-03-31T15:55:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.396+0000 7f9cd7c00640 1 -- start start 2024-03-31T15:55:15.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd006bc30 0x7f9cd00b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd7c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9cd015dff0 0x7f9cd015e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd7c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00d1b90 con 0x7f9cd006b850 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd7c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9cd0161e30 con 0x7f9cd015dff0 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd0162110 con 0x7f9cd006bc30 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9cd006b850 0x7f9cd00b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52278/0 (socket says 172.21.15.90:52278) 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5200640 1 -- 172.21.15.90:0/3473565027 learned_addr learned my addr 172.21.15.90:0/3473565027 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 --2- 172.21.15.90:0/3473565027 >> v2:172.21.15.90:3300/0 conn(0x7f9cd015dff0 0x7f9cd015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9ccfe00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd006bc30 0x7f9cd00b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9ccf400640 1 -- 172.21.15.90:0/3473565027 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 87052117 0 0) 0x7f9cd00d1b90 con 0x7f9cd006b850 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9ccf400640 1 -- 172.21.15.90:0/3473565027 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cb4003530 con 0x7f9cd006b850 2024-03-31T15:55:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 -- 172.21.15.90:0/3473565027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd006bc30 msgr2=0x7f9cd00b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd006bc30 0x7f9cd00b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 -- 172.21.15.90:0/3473565027 >> v1:172.21.15.90:6789/0 conn(0x7f9cd006b850 legacy=0x7f9cd00b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 -- 172.21.15.90:0/3473565027 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd01623f0 con 0x7f9cd015dff0 2024-03-31T15:55:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.400+0000 7f9cd5c00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd015dff0 0x7f9cd015e460 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80097c0 tx=0x7f9cc801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf0d1cc8efc1f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccf400640 1 -- 172.21.15.90:0/3473565027 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80b0070 con 0x7f9cd015dff0 2024-03-31T15:55:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccf400640 1 -- 172.21.15.90:0/3473565027 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc80ab600 con 0x7f9cd015dff0 2024-03-31T15:55:15.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccf400640 1 -- 172.21.15.90:0/3473565027 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80ab9d0 con 0x7f9cd015dff0 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- 172.21.15.90:0/3473565027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd015dff0 msgr2=0x7f9cd015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd015dff0 0x7f9cd015e460 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80097c0 tx=0x7f9cc801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- 172.21.15.90:0/3473565027 shutdown_connections 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd015dff0 0x7f9cd015e460 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/3473565027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd006bc30 0x7f9cd00b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- 172.21.15.90:0/3473565027 >> 172.21.15.90:0/3473565027 conn(0x7f9cd006c700 msgr2=0x7f9cd006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:15.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- 172.21.15.90:0/3473565027 shutdown_connections 2024-03-31T15:55:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- 172.21.15.90:0/3473565027 wait complete. 2024-03-31T15:55:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 Processor -- start 2024-03-31T15:55:15.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- start start 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9cd006b850 0x7f9cd01504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd015dff0 0x7f9cd0158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd015f330 con 0x7f9cd006bc30 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd015f1b0 con 0x7f9cd006b850 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd7c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd015f4b0 con 0x7f9cd015dff0 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56572/0 (socket says 172.21.15.90:56572) 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9ccfe00640 1 -- 172.21.15.90:0/648645590 learned_addr learned my addr 172.21.15.90:0/648645590 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd5200640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9cd006b850 0x7f9cd01504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.404+0000 7f9cd5c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd015dff0 0x7f9cd0158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 -- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd015dff0 msgr2=0x7f9cd0158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd015dff0 0x7f9cd0158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 -- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9cd006b850 msgr2=0x7f9cd01504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9cd006b850 0x7f9cd01504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 -- 172.21.15.90:0/648645590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd015c390 con 0x7f9cd006bc30 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9ccfe00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00e960 tx=0x7f9cbc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc017070 con 0x7f9cd006bc30 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc00ca00 con 0x7f9cd006bc30 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00cdb0 con 0x7f9cd006bc30 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd015c670 con 0x7f9cd006bc30 2024-03-31T15:55:15.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd015cc30 con 0x7f9cd006bc30 2024-03-31T15:55:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd00686d0 con 0x7f9cd006bc30 2024-03-31T15:55:15.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9cbc01a070 con 0x7f9cd006bc30 2024-03-31T15:55:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 0x7f9cb006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f9cbc086ba0 con 0x7f9cd006bc30 2024-03-31T15:55:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.408+0000 7f9cd5200640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 0x7f9cb006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:15.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.412+0000 7f9cd5200640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 0x7f9cb006a840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0005c10 tx=0x7f9cc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:15.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.412+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9cbc056c30 con 0x7f9cd006bc30 2024-03-31T15:55:15.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:15.572+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f9cd0072b80 con 0x7f9cd006bc30 2024-03-31T15:55:16.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.528+0000 7f9cccc00640 1 -- 172.21.15.90:0/648645590 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f9cbc05af30 con 0x7f9cd006bc30 2024-03-31T15:55:16.523 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 msgr2=0x7f9cb006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 0x7f9cb006a840 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0005c10 tx=0x7f9cc0009040 comp rx=0 tx=0).stop 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 msgr2=0x7f9cd01587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00e960 tx=0x7f9cbc00ee20 comp rx=0 tx=0).stop 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 shutdown_connections 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9cb0068380 0x7f9cb006a840 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9cd015dff0 0x7f9cd0158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9cd006bc30 0x7f9cd01587d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 --2- 172.21.15.90:0/648645590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9cd006b850 0x7f9cd01504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 >> 172.21.15.90:0/648645590 conn(0x7f9cd006c700 msgr2=0x7f9cd0161080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 shutdown_connections 2024-03-31T15:55:16.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.532+0000 7f9cd7c00640 1 -- 172.21.15.90:0/648645590 wait complete. 2024-03-31T15:55:16.554 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:55:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 Processor -- start 2024-03-31T15:55:16.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 -- start start 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f58dc15e250 0x7f58dc06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58dc15e630 con 0x7f58dc06f730 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58dc06b840 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.704+0000 7f58e3800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f58dc0b5920 con 0x7f58dc15e250 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58d3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f58dc06f730 0x7f58dc0b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52286/0 (socket says 172.21.15.90:52286) 2024-03-31T15:55:16.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58d3e00640 1 -- 172.21.15.90:0/1168586622 learned_addr learned my addr 172.21.15.90:0/1168586622 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58e0e00640 1 --2- 172.21.15.90:0/1168586622 >> v2:172.21.15.102:3300/0 conn(0x7f58dc15e250 0x7f58dc06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268507022 0 0) 0x7f58dc15e630 con 0x7f58dc06f730 2024-03-31T15:55:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58c0003530 con 0x7f58dc06f730 2024-03-31T15:55:16.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221035141 0 0) 0x7f58dc06b840 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58dc15e630 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58e0e00640 1 -- 172.21.15.90:0/1168586622 >> v1:172.21.15.102:6789/0 conn(0x7f58dc0b3cf0 legacy=0x7f58dc0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58e0e00640 1 -- 172.21.15.90:0/1168586622 >> v1:172.21.15.90:6789/0 conn(0x7f58dc06f730 legacy=0x7f58dc0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58e0e00640 1 -- 172.21.15.90:0/1168586622 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc06bb20 con 0x7f58dc15e250 2024-03-31T15:55:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.708+0000 7f58e0e00640 1 --2- 172.21.15.90:0/1168586622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc06c8e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0097c0 tx=0x7f58cc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56b31d6bb7d624c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0b1070 con 0x7f58dc15e250 2024-03-31T15:55:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58cc0ac5f0 con 0x7f58dc15e250 2024-03-31T15:55:16.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58dbe00640 1 -- 172.21.15.90:0/1168586622 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0ac9c0 con 0x7f58dc15e250 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- 172.21.15.90:0/1168586622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 msgr2=0x7f58dc06c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 --2- 172.21.15.90:0/1168586622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc06c8e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0097c0 tx=0x7f58cc01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- 172.21.15.90:0/1168586622 shutdown_connections 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 --2- 172.21.15.90:0/1168586622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc06c8e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- 172.21.15.90:0/1168586622 >> 172.21.15.90:0/1168586622 conn(0x7f58dc0d05f0 msgr2=0x7f58dc0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- 172.21.15.90:0/1168586622 shutdown_connections 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- 172.21.15.90:0/1168586622 wait complete. 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 Processor -- start 2024-03-31T15:55:16.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- start start 2024-03-31T15:55:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58dc06f730 0x7f58dc0cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:16.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc06ce20 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc16c990 con 0x7f58dc15e250 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e3800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f58dc15a4f0 con 0x7f58dc06f730 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58d3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56586/0 (socket says 172.21.15.90:56586) 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58d3e00640 1 -- 172.21.15.90:0/3115227813 learned_addr learned my addr 172.21.15.90:0/3115227813 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e0e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58dc06f730 0x7f58dc0cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.712+0000 7f58e1800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 -- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58dc06f730 msgr2=0x7f58dc0cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58dc06f730 0x7f58dc0cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 -- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 msgr2=0x7f58dc153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 -- 172.21.15.90:0/3115227813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc155130 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d3e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58c400c500 tx=0x7f58c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4017070 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c4005cf0 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4007330 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58dc0ccdd0 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58dc15fcb0 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58dc0685f0 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f58c401a070 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 0x7f58b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.716+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f58c4086120 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.720+0000 7f58e0e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 0x7f58b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:16.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.720+0000 7f58e0e00640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 0x7f58b806a840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0ae000 tx=0x7f58cc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:16.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.720+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f58c4021020 con 0x7f58dc0b3cf0 2024-03-31T15:55:16.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:16.880+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f58dc067550 con 0x7f58dc0b3cf0 2024-03-31T15:55:17.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58d9600640 1 -- 172.21.15.90:0/3115227813 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f58dc155130 con 0x7f58dc0b3cf0 2024-03-31T15:55:17.529 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-31T15:55:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 msgr2=0x7f58b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 0x7f58b806a840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f58cc0ae000 tx=0x7f58cc0a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 msgr2=0x7f58dc15aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58c400c500 tx=0x7f58c400c9c0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 shutdown_connections 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58b8068380 0x7f58b806a840 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58dc15e250 0x7f58dc153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58dc0b3cf0 0x7f58dc15aa80 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 --2- 172.21.15.90:0/3115227813 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58dc06f730 0x7f58dc0cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.536+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 >> 172.21.15.90:0/3115227813 conn(0x7f58dc0d05f0 msgr2=0x7f58dc0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.540+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 shutdown_connections 2024-03-31T15:55:17.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.540+0000 7f58e3800640 1 -- 172.21.15.90:0/3115227813 wait complete. 2024-03-31T15:55:17.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:55:17.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.716+0000 7f921b600640 1 Processor -- start 2024-03-31T15:55:17.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.716+0000 7f921b600640 1 -- start start 2024-03-31T15:55:17.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.716+0000 7f921b600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f921406f730 0x7f92140b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:17.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.716+0000 7f921b600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921415e630 con 0x7f921406ddb0 2024-03-31T15:55:17.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f921b600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921406b840 con 0x7f921415e250 2024-03-31T15:55:17.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f921b600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f92140b5920 con 0x7f921406f730 2024-03-31T15:55:17.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f921406f730 0x7f92140b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:17.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f921406f730 0x7f92140b35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56588/0 (socket says 172.21.15.90:56588) 2024-03-31T15:55:17.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 -- 172.21.15.90:0/816171432 learned_addr learned my addr 172.21.15.90:0/816171432 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:17.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 -- 172.21.15.90:0/816171432 >> v1:172.21.15.102:6789/0 conn(0x7f921415e250 legacy=0x7f921406c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:17.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 -- 172.21.15.90:0/816171432 >> v1:172.21.15.90:6789/0 conn(0x7f921406ddb0 legacy=0x7f92140ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 -- 172.21.15.90:0/816171432 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921406bb20 con 0x7f921406f730 2024-03-31T15:55:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213e00640 1 --2- 172.21.15.90:0/816171432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f92140b35e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6befec92f99dd74 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213400640 1 -- 172.21.15.90:0/816171432 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92000b0070 con 0x7f921406f730 2024-03-31T15:55:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213400640 1 -- 172.21.15.90:0/816171432 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92000ab600 con 0x7f921406f730 2024-03-31T15:55:17.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.720+0000 7f9213400640 1 -- 172.21.15.90:0/816171432 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92000ab9d0 con 0x7f921406f730 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- 172.21.15.90:0/816171432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 msgr2=0x7f92140b35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 --2- 172.21.15.90:0/816171432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f92140b35e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9200009870 tx=0x7f920001c520 comp rx=0 tx=0).stop 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- 172.21.15.90:0/816171432 shutdown_connections 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 --2- 172.21.15.90:0/816171432 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f92140b35e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- 172.21.15.90:0/816171432 >> 172.21.15.90:0/816171432 conn(0x7f92140d05f0 msgr2=0x7f92140d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- 172.21.15.90:0/816171432 shutdown_connections 2024-03-31T15:55:17.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- 172.21.15.90:0/816171432 wait complete. 2024-03-31T15:55:17.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 Processor -- start 2024-03-31T15:55:17.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- start start 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 0x7f9214161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 0x7f92140b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f92140b3f30 con 0x7f921406f730 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f921406b5f0 con 0x7f921406ddb0 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f921b600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f921416c990 con 0x7f921415e250 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56592/0 (socket says 172.21.15.90:56592) 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 -- 172.21.15.90:0/1618562204 learned_addr learned my addr 172.21.15.90:0/1618562204 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:17.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9218c00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 0x7f9214161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9219600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 0x7f92140b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 -- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 msgr2=0x7f92140b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 0x7f92140b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 -- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 msgr2=0x7f9214161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 0x7f9214161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9213e00640 1 -- 172.21.15.90:0/1618562204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92141558c0 con 0x7f921406f730 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9218c00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 0x7f9214161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.724+0000 7f9219600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 0x7f92140b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9213e00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f920001abd0 tx=0x7f920001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92000b0040 con 0x7f921406f730 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92000b6420 con 0x7f921406f730 2024-03-31T15:55:17.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92000b6810 con 0x7f921406f730 2024-03-31T15:55:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9214161700 con 0x7f921406f730 2024-03-31T15:55:17.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921415f300 con 0x7f921406f730 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92140685f0 con 0x7f921406f730 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f92000be030 con 0x7f921406f730 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 0x7f91f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f9200120610 con 0x7f921406f730 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.728+0000 7f9218c00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 0x7f91f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:17.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.732+0000 7f9218c00640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 0x7f91f406a840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f920800a850 tx=0x7f9208008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:17.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.732+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f92000c5060 con 0x7f921406f730 2024-03-31T15:55:17.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:55:17.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:55:17.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:17.928+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9214067550 con 0x7f921406f730 2024-03-31T15:55:18.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.536+0000 7f9210c00640 1 -- 172.21.15.90:0/1618562204 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f9200016110 con 0x7f921406f730 2024-03-31T15:55:18.531 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 5 and data pool 6 2024-03-31T15:55:18.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.540+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 msgr2=0x7f91f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.540+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 0x7f91f406a840 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f920800a850 tx=0x7f9208008040 comp rx=0 tx=0).stop 2024-03-31T15:55:18.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.540+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 msgr2=0x7f9214154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.540+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f920001abd0 tx=0x7f920001ad60 comp rx=0 tx=0).stop 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 shutdown_connections 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f91f4068380 0x7f91f406a840 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f921415e250 0x7f92140b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f921406f730 0x7f9214154ab0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 --2- 172.21.15.90:0/1618562204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f921406ddb0 0x7f9214161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 >> 172.21.15.90:0/1618562204 conn(0x7f92140d05f0 msgr2=0x7f92140b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:18.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 shutdown_connections 2024-03-31T15:55:18.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.544+0000 7f921b600640 1 -- 172.21.15.90:0/1618562204 wait complete. 2024-03-31T15:55:18.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:18.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 Processor -- start 2024-03-31T15:55:18.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 -- start start 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd86c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd86c06d5f0 0x7fd86c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd86c15f780 con 0x7fd86c15e1b0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd86c06daa0 con 0x7fd86c06d5f0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.720+0000 7fd870e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd86c1622b0 con 0x7fd86c06c7b0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd86c15e1b0 0x7fd86c162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52312/0 (socket says 172.21.15.90:52312) 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869e00640 1 -- 172.21.15.90:0/399961770 learned_addr learned my addr 172.21.15.90:0/399961770 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd86c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd86a800640 1 --2- 172.21.15.90:0/399961770 >> v2:172.21.15.102:3300/0 conn(0x7fd86c06d5f0 0x7fd86c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657258149 0 0) 0x7fd86c15f780 con 0x7fd86c15e1b0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd850003530 con 0x7fd86c15e1b0 2024-03-31T15:55:18.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1062027870 0 0) 0x7fd850003530 con 0x7fd86c15e1b0 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 msgr2=0x7fd86c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd86c06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 >> v2:172.21.15.102:3300/0 conn(0x7fd86c06d5f0 msgr2=0x7fd86c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 --2- 172.21.15.90:0/399961770 >> v2:172.21.15.102:3300/0 conn(0x7fd86c06d5f0 0x7fd86c15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c15f1c0 con 0x7fd86c15e1b0 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fd85c001680 con 0x7fd86c15e1b0 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 >> v1:172.21.15.90:6789/0 conn(0x7fd86c15e1b0 legacy=0x7fd86c162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd8500039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd850003f40 0x7fd850004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850004d50 0x7fd8500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c06daa0 con 0x7fd850003f40 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c1622b0 con 0x7fd850004d50 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd85000ba20 con 0x7fd86c06c7b0 2024-03-31T15:55:18.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd850003f40 0x7fd850004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd869e00640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd8500039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd86a800640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850004d50 0x7fd8500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 -- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 msgr2=0x7fd8500039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c06c7b0 0x7fd8500039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 -- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850004d50 msgr2=0x7fd8500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850004d50 0x7fd8500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.724+0000 7fd861400640 1 -- 172.21.15.90:0/399961770 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd850003530 con 0x7fd850003f40 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd861400640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd850003f40 0x7fd850004810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd854009ae0 tx=0x7fd85401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850a94cd5f363e44 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8540096b0 con 0x7fd850003f40 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8540a64f0 con 0x7fd850003f40 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd869400640 1 -- 172.21.15.90:0/399961770 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8540a68e0 con 0x7fd850003f40 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- 172.21.15.90:0/399961770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd850003f40 msgr2=0x7fd850004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 --2- 172.21.15.90:0/399961770 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd850003f40 0x7fd850004810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd854009ae0 tx=0x7fd85401ca00 comp rx=0 tx=0).stop 2024-03-31T15:55:18.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd86a800640 1 -- 172.21.15.90:0/399961770 reap_dead start 2024-03-31T15:55:18.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- 172.21.15.90:0/399961770 shutdown_connections 2024-03-31T15:55:18.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- 172.21.15.90:0/399961770 >> 172.21.15.90:0/399961770 conn(0x7fd86c09aa40 msgr2=0x7fd86c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:18.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- 172.21.15.90:0/399961770 shutdown_connections 2024-03-31T15:55:18.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- 172.21.15.90:0/399961770 wait complete. 2024-03-31T15:55:18.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 Processor -- start 2024-03-31T15:55:18.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- start start 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850003f40 0x7fd86c159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c0afa00 0x7fd86c163710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c0d2940 con 0x7fd86c0b4b40 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd86c0d27c0 con 0x7fd850003f40 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd870e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd86c0d2ac0 con 0x7fd86c0afa00 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.728+0000 7fd861400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56612/0 (socket says 172.21.15.90:56612) 2024-03-31T15:55:18.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 -- 172.21.15.90:0/3859690661 learned_addr learned my addr 172.21.15.90:0/3859690661 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd869e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850003f40 0x7fd86c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd86a800640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c0afa00 0x7fd86c163710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 -- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c0afa00 msgr2=0x7fd86c163710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c0afa00 0x7fd86c163710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 -- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850003f40 msgr2=0x7fd86c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850003f40 0x7fd86c159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 -- 172.21.15.90:0/3859690661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86c164390 con 0x7fd86c0b4b40 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd861400640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd85401c830 tx=0x7fd850005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8540be020 con 0x7fd86c0b4b40 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8540a66a0 con 0x7fd86c0b4b40 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8540a6a90 con 0x7fd86c0b4b40 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86c164670 con 0x7fd86c0b4b40 2024-03-31T15:55:18.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86c1598d0 con 0x7fd86c0b4b40 2024-03-31T15:55:18.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd86c0afe10 con 0x7fd86c0b4b40 2024-03-31T15:55:18.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8540c1020 con 0x7fd86c0b4b40 2024-03-31T15:55:18.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.732+0000 7fd862a00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 0x7fd84806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:18.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.736+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd854121440 con 0x7fd86c0b4b40 2024-03-31T15:55:18.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.736+0000 7fd869e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 0x7fd84806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:18.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.736+0000 7fd869e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 0x7fd84806a870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd86c15d700 tx=0x7fd85c007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:18.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.740+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd86c0afe10 con 0x7fd86c0b4b40 2024-03-31T15:55:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.896+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd86c0c6e10 con 0x7fd86c0b4b40 2024-03-31T15:55:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.896+0000 7fd862a00640 1 -- 172.21.15.90:0/3859690661 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7fd8540c8030 con 0x7fd86c0b4b40 2024-03-31T15:55:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:18.891 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":38,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:17.930287+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 msgr2=0x7fd84806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 0x7fd84806a870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd86c15d700 tx=0x7fd85c007900 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 msgr2=0x7fd86c0af4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd85401c830 tx=0x7fd850005840 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 shutdown_connections 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd8480683b0 0x7fd84806a870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86c0afa00 0x7fd86c163710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86c0b4b40 0x7fd86c0af4c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 --2- 172.21.15.90:0/3859690661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd850003f40 0x7fd86c159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 >> 172.21.15.90:0/3859690661 conn(0x7fd86c09aa40 msgr2=0x7fd86c162740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 shutdown_connections 2024-03-31T15:55:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:18.900+0000 7fd870e00640 1 -- 172.21.15.90:0/3859690661 wait complete. 2024-03-31T15:55:18.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:19.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 Processor -- start 2024-03-31T15:55:19.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 -- start start 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43ac0cc9d0 0x7f43ac142030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 0x7f43ac0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43ac0b34f0 con 0x7f43ac06b820 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f43ac0b3ab0 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b3c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac141ba0 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f43ac0cc9d0 0x7f43ac142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f43ac06b820 0x7f43ac0ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52326/0 (socket says 172.21.15.90:52326) 2024-03-31T15:55:19.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1c00640 1 -- 172.21.15.90:0/1166045197 learned_addr learned my addr 172.21.15.90:0/1166045197 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43abe00640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 0x7f43ac0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1200640 1 -- 172.21.15.90:0/1166045197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 msgr2=0x7f43ac0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1200640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 0x7f43ac0b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1200640 1 -- 172.21.15.90:0/1166045197 >> v1:172.21.15.90:6789/0 conn(0x7f43ac06b820 legacy=0x7f43ac0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43b1200640 1 -- 172.21.15.90:0/1166045197 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac0b37d0 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.060+0000 7f43abe00640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 0x7f43ac0b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:19.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43b1200640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac142030 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097c0 tx=0x7f43a001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fa6d66d860af76 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43ab400640 1 -- 172.21.15.90:0/1166045197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00ab600 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43ab400640 1 -- 172.21.15.90:0/1166045197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a00b0070 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43ab400640 1 -- 172.21.15.90:0/1166045197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00ab9c0 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43b3c00640 1 -- 172.21.15.90:0/1166045197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0cc9d0 msgr2=0x7f43ac142030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.064+0000 7f43b3c00640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac142030 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f43a00097c0 tx=0x7f43a001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- 172.21.15.90:0/1166045197 shutdown_connections 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac0ccdb0 0x7f43ac0b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 --2- 172.21.15.90:0/1166045197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac142030 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- 172.21.15.90:0/1166045197 >> 172.21.15.90:0/1166045197 conn(0x7f43ac0b1c80 msgr2=0x7f43ac0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:19.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- 172.21.15.90:0/1166045197 shutdown_connections 2024-03-31T15:55:19.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- 172.21.15.90:0/1166045197 wait complete. 2024-03-31T15:55:19.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 Processor -- start 2024-03-31T15:55:19.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- start start 2024-03-31T15:55:19.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac06b820 0x7f43ac072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac0b30c0 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac0cefc0 con 0x7f43ac0cc9d0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b3c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac0b3240 con 0x7f43ac06b820 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56624/0 (socket says 172.21.15.90:56624) 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 -- 172.21.15.90:0/2958039633 learned_addr learned my addr 172.21.15.90:0/2958039633 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43abe00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1200640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac06b820 0x7f43ac072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 -- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac06b820 msgr2=0x7f43ac072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac06b820 0x7f43ac072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 -- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43ac0cc9d0 msgr2=0x7f43ac1551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac1551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 -- 172.21.15.90:0/2958039633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac06e860 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43b1c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f43a400ca80 tx=0x7f43a400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.068+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4017070 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a400abd0 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac06eb40 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4012400 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac06f100 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ac0686d0 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f43a401a070 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43a8c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 0x7f438c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.072+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f43a4086610 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.076+0000 7f43b1200640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 0x7f438c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.076+0000 7f43b1200640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 0x7f438c06a840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f43a00af000 tx=0x7f43a00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.080+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f43a4056480 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.256+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43ac0b4aa0 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.256+0000 7f43a8c00640 1 -- 172.21.15.90:0/2958039633 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16919 (secure 0 0 0) 0x7f43a405a780 con 0x7f43ac0ccdb0 2024-03-31T15:55:19.251 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:19.251 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":38,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:17.930287+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 msgr2=0x7f438c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 0x7f438c06a840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f43a00af000 tx=0x7f43a00a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 msgr2=0x7f43ac15f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f43a400ca80 tx=0x7f43a400cf40 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 shutdown_connections 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f438c068380 0x7f438c06a840 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43ac0ccdb0 0x7f43ac15f9a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43ac0cc9d0 0x7f43ac1551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 --2- 172.21.15.90:0/2958039633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43ac06b820 0x7f43ac072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 >> 172.21.15.90:0/2958039633 conn(0x7f43ac0b1c80 msgr2=0x7f43ac0aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 shutdown_connections 2024-03-31T15:55:19.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.260+0000 7f43b3c00640 1 -- 172.21.15.90:0/2958039633 wait complete. 2024-03-31T15:55:19.272 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:55:19.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.416+0000 7fa45e600640 1 Processor -- start 2024-03-31T15:55:19.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.416+0000 7fa45e600640 1 -- start start 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa45e600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa45e600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4580cd640 0x7fa4580af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa45e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4580cefc0 con 0x7fa45815c320 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa45e600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa4580aed70 con 0x7fa4580cd640 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa45e600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa4580cf2a0 con 0x7fa4580d2b20 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4580cd640 0x7fa4580af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa45815c320 0x7fa4580cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:52946/0 (socket says 172.21.15.90:52946) 2024-03-31T15:55:19.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4580cd640 0x7fa4580af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:33106/0 (socket says 172.21.15.90:33106) 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457400640 1 -- 172.21.15.90:0/4149322969 learned_addr learned my addr 172.21.15.90:0/4149322969 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa44fe00640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 -- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 msgr2=0x7fa4580cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 -- 172.21.15.90:0/4149322969 >> v1:172.21.15.102:6789/0 conn(0x7fa45815c320 legacy=0x7fa4580cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 -- 172.21.15.90:0/4149322969 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580cf580 con 0x7fa4580cd640 2024-03-31T15:55:19.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.420+0000 7fa457e00640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa4580af200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa450009d90 tx=0x7fa45001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7228e6a36dda131b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa456a00640 1 -- 172.21.15.90:0/4149322969 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500b2070 con 0x7fa4580cd640 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa456a00640 1 -- 172.21.15.90:0/4149322969 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4580d0f20 con 0x7fa4580cd640 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa456a00640 1 -- 172.21.15.90:0/4149322969 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500a5ad0 con 0x7fa4580cd640 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 msgr2=0x7fa4580af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa4580af200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa450009d90 tx=0x7fa45001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- 172.21.15.90:0/4149322969 shutdown_connections 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa4580af200 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 --2- 172.21.15.90:0/4149322969 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- 172.21.15.90:0/4149322969 >> 172.21.15.90:0/4149322969 conn(0x7fa4580d1f20 msgr2=0x7fa4580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- 172.21.15.90:0/4149322969 shutdown_connections 2024-03-31T15:55:19.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- 172.21.15.90:0/4149322969 wait complete. 2024-03-31T15:55:19.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 Processor -- start 2024-03-31T15:55:19.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.424+0000 7fa45e600640 1 -- start start 2024-03-31T15:55:19.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa458153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580b0360 con 0x7fa45815c320 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580b01e0 con 0x7fa4580cd640 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa4580b04e0 con 0x7fa4580d2b20 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56632/0 (socket says 172.21.15.90:56632) 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 -- 172.21.15.90:0/1846998242 learned_addr learned my addr 172.21.15.90:0/1846998242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457400640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa458153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa44fe00640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 -- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 msgr2=0x7fa4580b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580b6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 -- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 msgr2=0x7fa458153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa458153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 -- 172.21.15.90:0/1846998242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa458153bd0 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa457e00640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4500b0040 tx=0x7fa450019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500bd020 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4500b8470 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500b2040 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa458154430 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa458154150 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.428+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4580685f0 con 0x7fa45815c320 2024-03-31T15:55:19.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.432+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa4500c0020 con 0x7fa45815c320 2024-03-31T15:55:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.432+0000 7fa44f400640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 0x7fa43406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:19.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.432+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa4501218e0 con 0x7fa45815c320 2024-03-31T15:55:19.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.432+0000 7fa457400640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 0x7fa43406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:19.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.432+0000 7fa457400640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 0x7fa43406a840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa448005890 tx=0x7fa448009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:19.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.436+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa4500c7030 con 0x7fa45815c320 2024-03-31T15:55:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:19.628+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa45815b2c0 con 0x7fa45815c320 2024-03-31T15:55:20.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa44f400640 1 -- 172.21.15.90:0/1846998242 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7fa450016560 con 0x7fa45815c320 2024-03-31T15:55:20.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 msgr2=0x7fa43406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 0x7fa43406a840 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa448005890 tx=0x7fa448009040 comp rx=0 tx=0).stop 2024-03-31T15:55:20.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 msgr2=0x7fa4580b6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4500b0040 tx=0x7fa450019680 comp rx=0 tx=0).stop 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 shutdown_connections 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa434068380 0x7fa43406a840 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa45815c320 0x7fa4580b6b70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa4580d2b20 0x7fa4580b6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 --2- 172.21.15.90:0/1846998242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4580cd640 0x7fa458153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 >> 172.21.15.90:0/1846998242 conn(0x7fa4580d1f20 msgr2=0x7fa4580b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 shutdown_connections 2024-03-31T15:55:20.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.568+0000 7fa45e600640 1 -- 172.21.15.90:0/1846998242 wait complete. 2024-03-31T15:55:20.583 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:55:20.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:55:20.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.736+0000 7fa7b3e00640 1 Processor -- start 2024-03-31T15:55:20.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.736+0000 7fa7b3e00640 1 -- start start 2024-03-31T15:55:20.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa7b415e190 0x7fa7b40d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7b406d720 0x7fa7b40cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b40cc4f0 con 0x7fa7b406cb10 2024-03-31T15:55:20.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b40d1380 con 0x7fa7b406d720 2024-03-31T15:55:20.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b40cc7d0 con 0x7fa7b415e190 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7b406d720 0x7fa7b40cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7b406d720 0x7fa7b40cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56634/0 (socket says 172.21.15.90:56634) 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 -- 172.21.15.90:0/2923874671 learned_addr learned my addr 172.21.15.90:0/2923874671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b2a00640 1 --2- 172.21.15.90:0/2923874671 >> v2:172.21.15.102:3300/0 conn(0x7fa7b415e190 0x7fa7b40d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 -- 172.21.15.90:0/2923874671 >> v2:172.21.15.102:3300/0 conn(0x7fa7b415e190 msgr2=0x7fa7b40d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 --2- 172.21.15.90:0/2923874671 >> v2:172.21.15.102:3300/0 conn(0x7fa7b415e190 0x7fa7b40d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 -- 172.21.15.90:0/2923874671 >> v1:172.21.15.102:6789/0 conn(0x7fa7b406cb10 legacy=0x7fa7b406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.740+0000 7fa7b3400640 1 -- 172.21.15.90:0/2923874671 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40ccab0 con 0x7fa7b406d720 2024-03-31T15:55:20.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3400640 1 --2- 172.21.15.90:0/2923874671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406d720 0x7fa7b40cd2b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40cf870 tx=0x7fa7ac01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7242984d1b68d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:20.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b1600640 1 -- 172.21.15.90:0/2923874671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0a5810 con 0x7fa7b406d720 2024-03-31T15:55:20.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b1600640 1 -- 172.21.15.90:0/2923874671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7ac0a59b0 con 0x7fa7b406d720 2024-03-31T15:55:20.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b1600640 1 -- 172.21.15.90:0/2923874671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0a5d80 con 0x7fa7b406d720 2024-03-31T15:55:20.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- 172.21.15.90:0/2923874671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406d720 msgr2=0x7fa7b40cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/2923874671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406d720 0x7fa7b40cd2b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40cf870 tx=0x7fa7ac01c930 comp rx=0 tx=0).stop 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- 172.21.15.90:0/2923874671 shutdown_connections 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/2923874671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406d720 0x7fa7b40cd2b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/2923874671 >> v2:172.21.15.102:3300/0 conn(0x7fa7b415e190 0x7fa7b40d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- 172.21.15.90:0/2923874671 >> 172.21.15.90:0/2923874671 conn(0x7fa7b40d1d70 msgr2=0x7fa7b40d2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- 172.21.15.90:0/2923874671 shutdown_connections 2024-03-31T15:55:20.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- 172.21.15.90:0/2923874671 wait complete. 2024-03-31T15:55:20.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 Processor -- start 2024-03-31T15:55:20.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.744+0000 7fa7b3e00640 1 -- start start 2024-03-31T15:55:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 0x7fa7b415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 0x7fa7b415c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b415e6f0 con 0x7fa7b406cb10 2024-03-31T15:55:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b415e570 con 0x7fa7b406d720 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b416ae80 con 0x7fa7b415e190 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56650/0 (socket says 172.21.15.90:56650) 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 -- 172.21.15.90:0/1180288195 learned_addr learned my addr 172.21.15.90:0/1180288195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2000640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 0x7fa7b415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3400640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 0x7fa7b415c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 -- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 msgr2=0x7fa7b415c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 0x7fa7b415c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 -- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 msgr2=0x7fa7b415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 0x7fa7b415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 -- 172.21.15.90:0/1180288195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40b5690 con 0x7fa7b406cb10 2024-03-31T15:55:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2000640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 0x7fa7b415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3400640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 0x7fa7b415c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b2a00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400d620 tx=0x7fa7a400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400b620 con 0x7fa7b406cb10 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a4019070 con 0x7fa7b406cb10 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400ba20 con 0x7fa7b406cb10 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b40b5970 con 0x7fa7b406cb10 2024-03-31T15:55:20.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b40b5f30 con 0x7fa7b406cb10 2024-03-31T15:55:20.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b40685f0 con 0x7fa7b406cb10 2024-03-31T15:55:20.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.748+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7a400bbc0 con 0x7fa7b406cb10 2024-03-31T15:55:20.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.752+0000 7fa7aaa00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 0x7fa79406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:20.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.752+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa7a4086fe0 con 0x7fa7b406cb10 2024-03-31T15:55:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.752+0000 7fa7b2000640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 0x7fa79406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:20.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.752+0000 7fa7b2000640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 0x7fa79406a840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4168ba0 tx=0x7fa7a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:20.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.756+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa7a4057440 con 0x7fa7b406cb10 2024-03-31T15:55:20.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:20.948+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa7b415aed0 con 0x7fa7b406cb10 2024-03-31T15:55:21.506 INFO:tasks.ceph.mds.c.smithi090.stderr:2024-03-31T15:55:21.512+0000 7f7908c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:55:21.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.568+0000 7fa7aaa00640 1 -- 172.21.15.90:0/1180288195 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7fa7a4056d40 con 0x7fa7b406cb10 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 msgr2=0x7fa79406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 0x7fa79406a840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4168ba0 tx=0x7fa7a0009040 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 msgr2=0x7fa7b414d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400d620 tx=0x7fa7a400dae0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 shutdown_connections 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa794068380 0x7fa79406a840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa7b415e190 0x7fa7b415c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7b406d720 0x7fa7b415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 --2- 172.21.15.90:0/1180288195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7b406cb10 0x7fa7b414d0b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 >> 172.21.15.90:0/1180288195 conn(0x7fa7b40d1d70 msgr2=0x7fa7b40ae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:21.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 shutdown_connections 2024-03-31T15:55:21.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.572+0000 7fa7b3e00640 1 -- 172.21.15.90:0/1180288195 wait complete. 2024-03-31T15:55:21.585 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:55:21.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 Processor -- start 2024-03-31T15:55:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 -- start start 2024-03-31T15:55:21.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f96040ccaf0 0x7f960415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960415f1d0 con 0x7f960415bfb0 2024-03-31T15:55:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960415f4b0 con 0x7f9604072840 2024-03-31T15:55:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f96040cb1a0 con 0x7f96040ccaf0 2024-03-31T15:55:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f960415bfb0 0x7f9604072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52336/0 (socket says 172.21.15.90:52336) 2024-03-31T15:55:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603400640 1 -- 172.21.15.90:0/163056087 learned_addr learned my addr 172.21.15.90:0/163056087 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603e00640 1 --2- 172.21.15.90:0/163056087 >> v2:172.21.15.102:3300/0 conn(0x7f96040ccaf0 0x7f960415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208388974 0 0) 0x7f960415f1d0 con 0x7f960415bfb0 2024-03-31T15:55:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95e8003530 con 0x7f960415bfb0 2024-03-31T15:55:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255159742 0 0) 0x7f960415f4b0 con 0x7f9604072840 2024-03-31T15:55:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f960415f1d0 con 0x7f9604072840 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 324152629 0 0) 0x7f95e8003530 con 0x7f960415bfb0 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 >> v2:172.21.15.102:3300/0 conn(0x7f96040ccaf0 msgr2=0x7f960415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 --2- 172.21.15.90:0/163056087 >> v2:172.21.15.102:3300/0 conn(0x7f96040ccaf0 0x7f960415e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 >> v1:172.21.15.102:6789/0 conn(0x7f9604072840 legacy=0x7f960406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960415f790 con 0x7f960415bfb0 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f95f00016a0 con 0x7f960415bfb0 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603e00640 1 --2- 172.21.15.90:0/163056087 >> v2:172.21.15.102:3300/0 conn(0x7f96040ccaf0 0x7f960415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 >> v1:172.21.15.90:6789/0 conn(0x7f960415bfb0 legacy=0x7f9604072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f95e8003b50 0x7f95e8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e8004810 0x7f95e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f95e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f95e800c020 con 0x7f95e80051d0 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f95e800c1d0 con 0x7f95e8004810 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f95e800c3d0 con 0x7f95e8003b50 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f95e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603400640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e8004810 0x7f95e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9603e00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f95e8003b50 0x7f95e8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 -- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f95e8003b50 msgr2=0x7f95e8003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f95e8003b50 0x7f95e8003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 -- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e8004810 msgr2=0x7f95e8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e8004810 0x7f95e8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.728+0000 7f9609000640 1 -- 172.21.15.90:0/163056087 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e8003530 con 0x7f95e80051d0 2024-03-31T15:55:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609000640 1 -- 172.21.15.90:0/163056087 reap_dead start 2024-03-31T15:55:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609000640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f95e800b9d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f95fc001fd0 tx=0x7f95fc01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379db81ec8cfd24f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc001c20 con 0x7f95e80051d0 2024-03-31T15:55:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95fc003b80 con 0x7f95e80051d0 2024-03-31T15:55:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9602a00640 1 -- 172.21.15.90:0/163056087 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc0050d0 con 0x7f95e80051d0 2024-03-31T15:55:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 msgr2=0x7f95e800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f95e800b9d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f95fc001fd0 tx=0x7f95fc01c900 comp rx=0 tx=0).stop 2024-03-31T15:55:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- 172.21.15.90:0/163056087 shutdown_connections 2024-03-31T15:55:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 --2- 172.21.15.90:0/163056087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f95e800b9d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- 172.21.15.90:0/163056087 >> 172.21.15.90:0/163056087 conn(0x7f96040cd660 msgr2=0x7f96040cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- 172.21.15.90:0/163056087 shutdown_connections 2024-03-31T15:55:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- 172.21.15.90:0/163056087 wait complete. 2024-03-31T15:55:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 Processor -- start 2024-03-31T15:55:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- start start 2024-03-31T15:55:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96041577e0 0x7f9604146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 0x7f96041f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f96040cced0 con 0x7f95e80051d0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f960406f530 con 0x7f9604146bb0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.736+0000 7f9609a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9604145f00 con 0x7f96041577e0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56660/0 (socket says 172.21.15.90:56660) 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 -- 172.21.15.90:0/2857500120 learned_addr learned my addr 172.21.15.90:0/2857500120 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9609000640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 0x7f96041f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603400640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96041577e0 0x7f9604146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 -- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96041577e0 msgr2=0x7f9604146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96041577e0 0x7f9604146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 -- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 msgr2=0x7f96041f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 0x7f96041f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 -- 172.21.15.90:0/2857500120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96041f4ad0 con 0x7f95e80051d0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9609000640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 0x7f96041f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9603e00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95f4011ca0 tx=0x7f95f400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f400fd20 con 0x7f95e80051d0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f4004020 con 0x7f95e80051d0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f40043f0 con 0x7f95e80051d0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96040b3a20 con 0x7f95e80051d0 2024-03-31T15:55:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9604146150 con 0x7f95e80051d0 2024-03-31T15:55:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9604068d00 con 0x7f95e80051d0 2024-03-31T15:55:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.740+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f95f4012040 con 0x7f95e80051d0 2024-03-31T15:55:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.744+0000 7f95fbe00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 0x7f95e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.744+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f95f4087070 con 0x7f95e80051d0 2024-03-31T15:55:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.744+0000 7f9603400640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 0x7f95e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.744+0000 7f9603400640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 0x7f95e406a870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95f0001060 tx=0x7f95f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:21.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.744+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f95f4019030 con 0x7f95e80051d0 2024-03-31T15:55:21.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.936+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96040d0990 con 0x7f95e80051d0 2024-03-31T15:55:21.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.936+0000 7f95fbe00640 1 -- 172.21.15.90:0/2857500120 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5184 (secure 0 0 0) 0x7f95f4057490 con 0x7f95e80051d0 2024-03-31T15:55:21.930 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:21.931 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:20.953376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:55:21.931 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 msgr2=0x7f95e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 0x7f95e406a870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f95f0001060 tx=0x7f95f0009040 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 msgr2=0x7f96041e8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95f4011ca0 tx=0x7f95f400f610 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 shutdown_connections 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f95e40683b0 0x7f95e406a870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9604146bb0 0x7f96041f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96041577e0 0x7f9604146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 --2- 172.21.15.90:0/2857500120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f95e80051d0 0x7f96041e8ce0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 >> 172.21.15.90:0/2857500120 conn(0x7f96040cd660 msgr2=0x7f960406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 shutdown_connections 2024-03-31T15:55:21.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:21.940+0000 7f9609a00640 1 -- 172.21.15.90:0/2857500120 wait complete. 2024-03-31T15:55:21.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:22.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 Processor -- start 2024-03-31T15:55:22.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 -- start start 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 0x7f678c0d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f678c160350 con 0x7f678c15c120 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f678c160630 con 0x7f678c0cca30 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.092+0000 7f6791000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f678c160bf0 con 0x7f678c15fd00 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6789e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f678c15c120 0x7f678c0b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52338/0 (socket says 172.21.15.90:52338) 2024-03-31T15:55:22.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6789e00640 1 -- 172.21.15.90:0/3306586461 learned_addr learned my addr 172.21.15.90:0/3306586461 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6783e00640 1 --2- 172.21.15.90:0/3306586461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 0x7f678c0d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939552925 0 0) 0x7f678c160350 con 0x7f678c15c120 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6770003530 con 0x7f678c15c120 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746275180 0 0) 0x7f678c160630 con 0x7f678c0cca30 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6783e00640 1 -- 172.21.15.90:0/3306586461 >> v1:172.21.15.102:6789/0 conn(0x7f678c0cca30 legacy=0x7f678c0d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6783e00640 1 -- 172.21.15.90:0/3306586461 >> v1:172.21.15.90:6789/0 conn(0x7f678c15c120 legacy=0x7f678c0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6783e00640 1 -- 172.21.15.90:0/3306586461 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678c160910 con 0x7f678c15fd00 2024-03-31T15:55:22.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.096+0000 7f6783e00640 1 --2- 172.21.15.90:0/3306586461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 0x7f678c0d2680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f67740097f0 tx=0x7f677401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d8b19251851d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67740b0070 con 0x7f678c15fd00 2024-03-31T15:55:22.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67740ab600 con 0x7f678c15fd00 2024-03-31T15:55:22.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789400640 1 -- 172.21.15.90:0/3306586461 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67740ab9d0 con 0x7f678c15fd00 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- 172.21.15.90:0/3306586461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 msgr2=0x7f678c0d2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 --2- 172.21.15.90:0/3306586461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 0x7f678c0d2680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f67740097f0 tx=0x7f677401c450 comp rx=0 tx=0).stop 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- 172.21.15.90:0/3306586461 shutdown_connections 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 --2- 172.21.15.90:0/3306586461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15fd00 0x7f678c0d2680 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- 172.21.15.90:0/3306586461 >> 172.21.15.90:0/3306586461 conn(0x7f678c09aa40 msgr2=0x7f678c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- 172.21.15.90:0/3306586461 shutdown_connections 2024-03-31T15:55:22.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- 172.21.15.90:0/3306586461 wait complete. 2024-03-31T15:55:22.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 Processor -- start 2024-03-31T15:55:22.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- start start 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15c120 0x7f678c1668f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f678c15fd00 0x7f678c06c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f678c0d0d70 con 0x7f678c0cca30 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f678c0d0bf0 con 0x7f678c15fd00 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6791000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f678c0cce60 con 0x7f678c15c120 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56668/0 (socket says 172.21.15.90:56668) 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6789e00640 1 -- 172.21.15.90:0/348699386 learned_addr learned my addr 172.21.15.90:0/348699386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f6783e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15c120 0x7f678c1668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.100+0000 7f678a800640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f678c15fd00 0x7f678c06c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 -- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15c120 msgr2=0x7f678c1668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15c120 0x7f678c1668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 -- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f678c15fd00 msgr2=0x7f678c06c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f678c15fd00 0x7f678c06c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 -- 172.21.15.90:0/348699386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678c15e8d0 con 0x7f678c0cca30 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6789e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c450 tx=0x7f677c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c017070 con 0x7f678c0cca30 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f677c005cf0 con 0x7f678c0cca30 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678c165400 con 0x7f678c0cca30 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c0072f0 con 0x7f678c0cca30 2024-03-31T15:55:22.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f678c1659c0 con 0x7f678c0cca30 2024-03-31T15:55:22.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f678c0686d0 con 0x7f678c0cca30 2024-03-31T15:55:22.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.104+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f677c01a070 con 0x7f678c0cca30 2024-03-31T15:55:22.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.108+0000 7f6782000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 0x7f676806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.108+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f677c085ec0 con 0x7f678c0cca30 2024-03-31T15:55:22.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.108+0000 7f6783e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 0x7f676806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.108+0000 7f6783e00640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 0x7f676806a840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f67740ad000 tx=0x7f6774008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.112+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f677c055ca0 con 0x7f678c0cca30 2024-03-31T15:55:22.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f678c1666d0 con 0x7f678c0cca30 2024-03-31T15:55:22.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6782000640 1 -- 172.21.15.90:0/348699386 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16906 (secure 0 0 0) 0x7f677c059fa0 con 0x7f678c0cca30 2024-03-31T15:55:22.266 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:22.267 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":39,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:19.540370+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:22.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 msgr2=0x7f676806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 0x7f676806a840 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f67740ad000 tx=0x7f6774008040 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 msgr2=0x7f678c156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.272+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f677c00c450 tx=0x7f677c00c910 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 shutdown_connections 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6768068380 0x7f676806a840 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f678c15fd00 0x7f678c06c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f678c15c120 0x7f678c1668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 --2- 172.21.15.90:0/348699386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f678c0cca30 0x7f678c156460 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 >> 172.21.15.90:0/348699386 conn(0x7f678c09aa40 msgr2=0x7f678c15b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 shutdown_connections 2024-03-31T15:55:22.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.276+0000 7f6791000640 1 -- 172.21.15.90:0/348699386 wait complete. 2024-03-31T15:55:22.288 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:55:22.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:55:22.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.428+0000 7fdecbe00640 1 Processor -- start 2024-03-31T15:55:22.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.428+0000 7fdecbe00640 1 -- start start 2024-03-31T15:55:22.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdecc147760 0x7fdecc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdecc147b40 0x7fdecc161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecbe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdecc162720 con 0x7fdecc1620f0 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecbe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdecc0cee10 con 0x7fdecc147760 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdecc162a00 con 0x7fdecc147b40 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdecc147760 0x7fdecc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdecc147760 0x7fdecc0cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56678/0 (socket says 172.21.15.90:56678) 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 -- 172.21.15.90:0/2204574858 learned_addr learned my addr 172.21.15.90:0/2204574858 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdec3e00640 1 --2- 172.21.15.90:0/2204574858 >> v2:172.21.15.102:3300/0 conn(0x7fdecc147b40 0x7fdecc161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 -- 172.21.15.90:0/2204574858 >> v2:172.21.15.102:3300/0 conn(0x7fdecc147b40 msgr2=0x7fdecc161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:22.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 --2- 172.21.15.90:0/2204574858 >> v2:172.21.15.102:3300/0 conn(0x7fdecc147b40 0x7fdecc161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 -- 172.21.15.90:0/2204574858 >> v1:172.21.15.102:6789/0 conn(0x7fdecc1620f0 legacy=0x7fdecc06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.432+0000 7fdecaa00640 1 -- 172.21.15.90:0/2204574858 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdecc0ce7c0 con 0x7fdecc147760 2024-03-31T15:55:22.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecaa00640 1 --2- 172.21.15.90:0/2204574858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc0cf2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdecc06c6d0 tx=0x7fdebc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1f303334b11a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdeca000640 1 -- 172.21.15.90:0/2204574858 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0ab600 con 0x7fdecc147760 2024-03-31T15:55:22.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdeca000640 1 -- 172.21.15.90:0/2204574858 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdebc0b0070 con 0x7fdecc147760 2024-03-31T15:55:22.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdeca000640 1 -- 172.21.15.90:0/2204574858 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0ab9c0 con 0x7fdecc147760 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 -- 172.21.15.90:0/2204574858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 msgr2=0x7fdecc0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 --2- 172.21.15.90:0/2204574858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc0cf2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fdecc06c6d0 tx=0x7fdebc01c4f0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 -- 172.21.15.90:0/2204574858 shutdown_connections 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 --2- 172.21.15.90:0/2204574858 >> v2:172.21.15.102:3300/0 conn(0x7fdecc147b40 0x7fdecc161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 --2- 172.21.15.90:0/2204574858 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc0cf2a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 -- 172.21.15.90:0/2204574858 >> 172.21.15.90:0/2204574858 conn(0x7fdecc0cd660 msgr2=0x7fdecc0cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 -- 172.21.15.90:0/2204574858 shutdown_connections 2024-03-31T15:55:22.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.436+0000 7fdecbe00640 1 -- 172.21.15.90:0/2204574858 wait complete. 2024-03-31T15:55:22.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 Processor -- start 2024-03-31T15:55:22.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- start start 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 0x7fdecc143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 0x7fdecc144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdecc0cc8c0 con 0x7fdecc147760 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdecc06bf10 con 0x7fdecc147b40 2024-03-31T15:55:22.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdecc0cca40 con 0x7fdecc1620f0 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56680/0 (socket says 172.21.15.90:56680) 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 -- 172.21.15.90:0/3324476044 learned_addr learned my addr 172.21.15.90:0/3324476044 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdec3e00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 0x7fdecc143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecb400640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 0x7fdecc144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 -- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 msgr2=0x7fdecc144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 0x7fdecc144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 -- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 msgr2=0x7fdecc143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 0x7fdecc143d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 -- 172.21.15.90:0/3324476044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdecc1423d0 con 0x7fdecc147760 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdec3e00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 0x7fdecc143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecb400640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 0x7fdecc144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecaa00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0ad040 tx=0x7fdebc01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0be5f0 con 0x7fdecc147760 2024-03-31T15:55:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdebc0be790 con 0x7fdecc147760 2024-03-31T15:55:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0beb60 con 0x7fdecc147760 2024-03-31T15:55:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdecc1426b0 con 0x7fdecc147760 2024-03-31T15:55:22.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.440+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdecc142c70 con 0x7fdecc147760 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdecc068d00 con 0x7fdecc147760 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdebc0aa070 con 0x7fdecc147760 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdec3400640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 0x7fdea806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fdebc1209d0 con 0x7fdecc147760 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdec3e00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 0x7fdea806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.444+0000 7fdec3e00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 0x7fdea806a840 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdecc147180 tx=0x7fdeb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.452+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdebc016110 con 0x7fdecc147760 2024-03-31T15:55:22.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.608+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdecc161b40 con 0x7fdea8068380 2024-03-31T15:55:22.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.616+0000 7fdec3400640 1 -- 172.21.15.90:0/3324476044 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243097 (secure 0 0 0) 0x7fdecc161b40 con 0x7fdea8068380 2024-03-31T15:55:22.613 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:22.615 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 msgr2=0x7fdea806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 0x7fdea806a840 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdecc147180 tx=0x7fdeb4008040 comp rx=0 tx=0).stop 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 msgr2=0x7fdecc141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0ad040 tx=0x7fdebc01ad30 comp rx=0 tx=0).stop 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 shutdown_connections 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdea8068380 0x7fdea806a840 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdecc1620f0 0x7fdecc144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdecc147b40 0x7fdecc143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 --2- 172.21.15.90:0/3324476044 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdecc147760 0x7fdecc141020 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 >> 172.21.15.90:0/3324476044 conn(0x7fdecc0cd660 msgr2=0x7fdecc1612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:22.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 shutdown_connections 2024-03-31T15:55:22.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.620+0000 7fdecbe00640 1 -- 172.21.15.90:0/3324476044 wait complete. 2024-03-31T15:55:22.634 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-03-31T15:55:21.238712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":183,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":220572,"kb_used_data":4376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754148,"statfs":{"total":773094113280,"available":772868247552,"internally_reserved":0,"allocated":4481024,"data_stored":2565831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.704321"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548723+0000","last_change":"2024-03-31T15:55:18.569836+0000","last_active":"2024-03-31T15:55:19.548723+0000","last_peered":"2024-03-31T15:55:19.548723+0000","last_clean":"2024-03-31T15:55:19.548723+0000","last_became_active":"2024-03-31T15:55:18.569542+0000","last_became_peered":"2024-03-31T15:55:18.569542+0000","last_unstale":"2024-03-31T15:55:19.548723+0000","last_undegraded":"2024-03-31T15:55:19.548723+0000","last_fullsized":"2024-03-31T15:55:19.548723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:52.818181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548259+0000","last_change":"2024-03-31T15:55:18.570636+0000","last_active":"2024-03-31T15:55:19.548259+0000","last_peered":"2024-03-31T15:55:19.548259+0000","last_clean":"2024-03-31T15:55:19.548259+0000","last_became_active":"2024-03-31T15:55:18.570347+0000","last_became_peered":"2024-03-31T15:55:18.570347+0000","last_unstale":"2024-03-31T15:55:19.548259+0000","last_undegraded":"2024-03-31T15:55:19.548259+0000","last_fullsized":"2024-03-31T15:55:19.548259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550774+0000","last_change":"2024-03-31T15:55:18.558889+0000","last_active":"2024-03-31T15:55:19.550774+0000","last_peered":"2024-03-31T15:55:19.550774+0000","last_clean":"2024-03-31T15:55:19.550774+0000","last_became_active":"2024-03-31T15:55:18.558620+0000","last_became_peered":"2024-03-31T15:55:18.558620+0000","last_unstale":"2024-03-31T15:55:19.550774+0000","last_undegraded":"2024-03-31T15:55:19.550774+0000","last_fullsized":"2024-03-31T15:55:19.550774+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:16.723939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550456+0000","last_change":"2024-03-31T15:55:18.552183+0000","last_active":"2024-03-31T15:55:19.550456+0000","last_peered":"2024-03-31T15:55:19.550456+0000","last_clean":"2024-03-31T15:55:19.550456+0000","last_became_active":"2024-03-31T15:55:18.551886+0000","last_became_peered":"2024-03-31T15:55:18.551886+0000","last_unstale":"2024-03-31T15:55:19.550456+0000","last_undegraded":"2024-03-31T15:55:19.550456+0000","last_fullsized":"2024-03-31T15:55:19.550456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:53:07.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547884+0000","last_change":"2024-03-31T15:55:18.572173+0000","last_active":"2024-03-31T15:55:19.547884+0000","last_peered":"2024-03-31T15:55:19.547884+0000","last_clean":"2024-03-31T15:55:19.547884+0000","last_became_active":"2024-03-31T15:55:18.571909+0000","last_became_peered":"2024-03-31T15:55:18.571909+0000","last_unstale":"2024-03-31T15:55:19.547884+0000","last_undegraded":"2024-03-31T15:55:19.547884+0000","last_fullsized":"2024-03-31T15:55:19.547884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580155+0000","last_change":"2024-03-31T15:55:18.573440+0000","last_active":"2024-03-31T15:55:18.580155+0000","last_peered":"2024-03-31T15:55:18.580155+0000","last_clean":"2024-03-31T15:55:18.580155+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:18.580155+0000","last_undegraded":"2024-03-31T15:55:18.580155+0000","last_fullsized":"2024-03-31T15:55:18.580155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:44.338598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.579763+0000","last_change":"2024-03-31T15:55:18.571940+0000","last_active":"2024-03-31T15:55:18.579763+0000","last_peered":"2024-03-31T15:55:18.579763+0000","last_clean":"2024-03-31T15:55:18.579763+0000","last_became_active":"2024-03-31T15:55:18.571687+0000","last_became_peered":"2024-03-31T15:55:18.571687+0000","last_unstale":"2024-03-31T15:55:18.579763+0000","last_undegraded":"2024-03-31T15:55:18.579763+0000","last_fullsized":"2024-03-31T15:55:18.579763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:24.106111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550838+0000","last_change":"2024-03-31T15:55:18.565501+0000","last_active":"2024-03-31T15:55:19.550838+0000","last_peered":"2024-03-31T15:55:19.550838+0000","last_clean":"2024-03-31T15:55:19.550838+0000","last_became_active":"2024-03-31T15:55:18.565247+0000","last_became_peered":"2024-03-31T15:55:18.565247+0000","last_unstale":"2024-03-31T15:55:19.550838+0000","last_undegraded":"2024-03-31T15:55:19.550838+0000","last_fullsized":"2024-03-31T15:55:19.550838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:55.707922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583121+0000","last_change":"2024-03-31T15:55:18.575290+0000","last_active":"2024-03-31T15:55:18.583121+0000","last_peered":"2024-03-31T15:55:18.583121+0000","last_clean":"2024-03-31T15:55:18.583121+0000","last_became_active":"2024-03-31T15:55:18.574928+0000","last_became_peered":"2024-03-31T15:55:18.574928+0000","last_unstale":"2024-03-31T15:55:18.583121+0000","last_undegraded":"2024-03-31T15:55:18.583121+0000","last_fullsized":"2024-03-31T15:55:18.583121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:04.640612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550628+0000","last_change":"2024-03-31T15:55:18.563439+0000","last_active":"2024-03-31T15:55:19.550628+0000","last_peered":"2024-03-31T15:55:19.550628+0000","last_clean":"2024-03-31T15:55:19.550628+0000","last_became_active":"2024-03-31T15:55:18.563160+0000","last_became_peered":"2024-03-31T15:55:18.563160+0000","last_unstale":"2024-03-31T15:55:19.550628+0000","last_undegraded":"2024-03-31T15:55:19.550628+0000","last_fullsized":"2024-03-31T15:55:19.550628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:33.025431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583094+0000","last_change":"2024-03-31T15:55:18.574506+0000","last_active":"2024-03-31T15:55:18.583094+0000","last_peered":"2024-03-31T15:55:18.583094+0000","last_clean":"2024-03-31T15:55:18.583094+0000","last_became_active":"2024-03-31T15:55:18.570090+0000","last_became_peered":"2024-03-31T15:55:18.570090+0000","last_unstale":"2024-03-31T15:55:18.583094+0000","last_undegraded":"2024-03-31T15:55:18.583094+0000","last_fullsized":"2024-03-31T15:55:18.583094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:14.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580312+0000","last_change":"2024-03-31T15:55:18.570263+0000","last_active":"2024-03-31T15:55:18.580312+0000","last_peered":"2024-03-31T15:55:18.580312+0000","last_clean":"2024-03-31T15:55:18.580312+0000","last_became_active":"2024-03-31T15:55:18.569923+0000","last_became_peered":"2024-03-31T15:55:18.569923+0000","last_unstale":"2024-03-31T15:55:18.580312+0000","last_undegraded":"2024-03-31T15:55:18.580312+0000","last_fullsized":"2024-03-31T15:55:18.580312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:29.731970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548666+0000","last_change":"2024-03-31T15:55:18.583489+0000","last_active":"2024-03-31T15:55:19.548666+0000","last_peered":"2024-03-31T15:55:19.548666+0000","last_clean":"2024-03-31T15:55:19.548666+0000","last_became_active":"2024-03-31T15:55:18.583246+0000","last_became_peered":"2024-03-31T15:55:18.583246+0000","last_unstale":"2024-03-31T15:55:19.548666+0000","last_undegraded":"2024-03-31T15:55:19.548666+0000","last_fullsized":"2024-03-31T15:55:19.548666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:48:46.491057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.577287+0000","last_change":"2024-03-31T15:55:17.569775+0000","last_active":"2024-03-31T15:55:18.577287+0000","last_peered":"2024-03-31T15:55:18.577287+0000","last_clean":"2024-03-31T15:55:18.577287+0000","last_became_active":"2024-03-31T15:55:17.568127+0000","last_became_peered":"2024-03-31T15:55:17.568127+0000","last_unstale":"2024-03-31T15:55:18.577287+0000","last_undegraded":"2024-03-31T15:55:18.577287+0000","last_fullsized":"2024-03-31T15:55:18.577287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:19:15.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548413+0000","last_change":"2024-03-31T15:55:18.572911+0000","last_active":"2024-03-31T15:55:19.548413+0000","last_peered":"2024-03-31T15:55:19.548413+0000","last_clean":"2024-03-31T15:55:19.548413+0000","last_became_active":"2024-03-31T15:55:18.572663+0000","last_became_peered":"2024-03-31T15:55:18.572663+0000","last_unstale":"2024-03-31T15:55:19.548413+0000","last_undegraded":"2024-03-31T15:55:19.548413+0000","last_fullsized":"2024-03-31T15:55:19.548413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550725+0000","last_change":"2024-03-31T15:55:18.558312+0000","last_active":"2024-03-31T15:55:19.550725+0000","last_peered":"2024-03-31T15:55:19.550725+0000","last_clean":"2024-03-31T15:55:19.550725+0000","last_became_active":"2024-03-31T15:55:18.556133+0000","last_became_peered":"2024-03-31T15:55:18.556133+0000","last_unstale":"2024-03-31T15:55:19.550725+0000","last_undegraded":"2024-03-31T15:55:19.550725+0000","last_fullsized":"2024-03-31T15:55:19.550725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:51:25.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548747+0000","last_change":"2024-03-31T15:55:17.559251+0000","last_active":"2024-03-31T15:55:19.548747+0000","last_peered":"2024-03-31T15:55:19.548747+0000","last_clean":"2024-03-31T15:55:19.548747+0000","last_became_active":"2024-03-31T15:55:17.558977+0000","last_became_peered":"2024-03-31T15:55:17.558977+0000","last_unstale":"2024-03-31T15:55:19.548747+0000","last_undegraded":"2024-03-31T15:55:19.548747+0000","last_fullsized":"2024-03-31T15:55:19.548747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:23.098269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583255+0000","last_change":"2024-03-31T15:55:18.576163+0000","last_active":"2024-03-31T15:55:18.583255+0000","last_peered":"2024-03-31T15:55:18.583255+0000","last_clean":"2024-03-31T15:55:18.583255+0000","last_became_active":"2024-03-31T15:55:18.575491+0000","last_became_peered":"2024-03-31T15:55:18.575491+0000","last_unstale":"2024-03-31T15:55:18.583255+0000","last_undegraded":"2024-03-31T15:55:18.583255+0000","last_fullsized":"2024-03-31T15:55:18.583255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:25.287778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548239+0000","last_change":"2024-03-31T15:55:17.561703+0000","last_active":"2024-03-31T15:55:19.548239+0000","last_peered":"2024-03-31T15:55:19.548239+0000","last_clean":"2024-03-31T15:55:19.548239+0000","last_became_active":"2024-03-31T15:55:17.561448+0000","last_became_peered":"2024-03-31T15:55:17.561448+0000","last_unstale":"2024-03-31T15:55:19.548239+0000","last_undegraded":"2024-03-31T15:55:19.548239+0000","last_fullsized":"2024-03-31T15:55:19.548239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:38.684847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548566+0000","last_change":"2024-03-31T15:55:17.565178+0000","last_active":"2024-03-31T15:55:19.548566+0000","last_peered":"2024-03-31T15:55:19.548566+0000","last_clean":"2024-03-31T15:55:19.548566+0000","last_became_active":"2024-03-31T15:55:17.564972+0000","last_became_peered":"2024-03-31T15:55:17.564972+0000","last_unstale":"2024-03-31T15:55:19.548566+0000","last_undegraded":"2024-03-31T15:55:19.548566+0000","last_fullsized":"2024-03-31T15:55:19.548566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:43.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548114+0000","last_change":"2024-03-31T15:55:18.576538+0000","last_active":"2024-03-31T15:55:19.548114+0000","last_peered":"2024-03-31T15:55:19.548114+0000","last_clean":"2024-03-31T15:55:19.548114+0000","last_became_active":"2024-03-31T15:55:18.569912+0000","last_became_peered":"2024-03-31T15:55:18.569912+0000","last_unstale":"2024-03-31T15:55:19.548114+0000","last_undegraded":"2024-03-31T15:55:19.548114+0000","last_fullsized":"2024-03-31T15:55:19.548114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580350+0000","last_change":"2024-03-31T15:55:18.567828+0000","last_active":"2024-03-31T15:55:18.580350+0000","last_peered":"2024-03-31T15:55:18.580350+0000","last_clean":"2024-03-31T15:55:18.580350+0000","last_became_active":"2024-03-31T15:55:18.567508+0000","last_became_peered":"2024-03-31T15:55:18.567508+0000","last_unstale":"2024-03-31T15:55:18.580350+0000","last_undegraded":"2024-03-31T15:55:18.580350+0000","last_fullsized":"2024-03-31T15:55:18.580350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:39.692689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548449+0000","last_change":"2024-03-31T15:55:18.570563+0000","last_active":"2024-03-31T15:55:19.548449+0000","last_peered":"2024-03-31T15:55:19.548449+0000","last_clean":"2024-03-31T15:55:19.548449+0000","last_became_active":"2024-03-31T15:55:18.570292+0000","last_became_peered":"2024-03-31T15:55:18.570292+0000","last_unstale":"2024-03-31T15:55:19.548449+0000","last_undegraded":"2024-03-31T15:55:19.548449+0000","last_fullsized":"2024-03-31T15:55:19.548449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.584994+0000","last_change":"2024-03-31T15:55:17.568544+0000","last_active":"2024-03-31T15:55:18.584994+0000","last_peered":"2024-03-31T15:55:18.584994+0000","last_clean":"2024-03-31T15:55:18.584994+0000","last_became_active":"2024-03-31T15:55:17.567942+0000","last_became_peered":"2024-03-31T15:55:17.567942+0000","last_unstale":"2024-03-31T15:55:18.584994+0000","last_undegraded":"2024-03-31T15:55:18.584994+0000","last_fullsized":"2024-03-31T15:55:18.584994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:10.013322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548171+0000","last_change":"2024-03-31T15:55:18.574899+0000","last_active":"2024-03-31T15:55:19.548171+0000","last_peered":"2024-03-31T15:55:19.548171+0000","last_clean":"2024-03-31T15:55:19.548171+0000","last_became_active":"2024-03-31T15:55:18.574347+0000","last_became_peered":"2024-03-31T15:55:18.574347+0000","last_unstale":"2024-03-31T15:55:19.548171+0000","last_undegraded":"2024-03-31T15:55:19.548171+0000","last_fullsized":"2024-03-31T15:55:19.548171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550772+0000","last_change":"2024-03-31T15:55:17.562464+0000","last_active":"2024-03-31T15:55:19.550772+0000","last_peered":"2024-03-31T15:55:19.550772+0000","last_clean":"2024-03-31T15:55:19.550772+0000","last_became_active":"2024-03-31T15:55:17.562192+0000","last_became_peered":"2024-03-31T15:55:17.562192+0000","last_unstale":"2024-03-31T15:55:19.550772+0000","last_undegraded":"2024-03-31T15:55:19.550772+0000","last_fullsized":"2024-03-31T15:55:19.550772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:13.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550931+0000","last_change":"2024-03-31T15:55:18.572667+0000","last_active":"2024-03-31T15:55:19.550931+0000","last_peered":"2024-03-31T15:55:19.550931+0000","last_clean":"2024-03-31T15:55:19.550931+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.550931+0000","last_undegraded":"2024-03-31T15:55:19.550931+0000","last_fullsized":"2024-03-31T15:55:19.550931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:54:13.587576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583198+0000","last_change":"2024-03-31T15:55:18.573739+0000","last_active":"2024-03-31T15:55:18.583198+0000","last_peered":"2024-03-31T15:55:18.583198+0000","last_clean":"2024-03-31T15:55:18.583198+0000","last_became_active":"2024-03-31T15:55:18.573288+0000","last_became_peered":"2024-03-31T15:55:18.573288+0000","last_unstale":"2024-03-31T15:55:18.583198+0000","last_undegraded":"2024-03-31T15:55:18.583198+0000","last_fullsized":"2024-03-31T15:55:18.583198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:27.032794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550590+0000","last_change":"2024-03-31T15:55:17.561760+0000","last_active":"2024-03-31T15:55:19.550590+0000","last_peered":"2024-03-31T15:55:19.550590+0000","last_clean":"2024-03-31T15:55:19.550590+0000","last_became_active":"2024-03-31T15:55:17.561471+0000","last_became_peered":"2024-03-31T15:55:17.561471+0000","last_unstale":"2024-03-31T15:55:19.550590+0000","last_undegraded":"2024-03-31T15:55:19.550590+0000","last_fullsized":"2024-03-31T15:55:19.550590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:31:27.557663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550455+0000","last_change":"2024-03-31T15:55:17.567522+0000","last_active":"2024-03-31T15:55:19.550455+0000","last_peered":"2024-03-31T15:55:19.550455+0000","last_clean":"2024-03-31T15:55:19.550455+0000","last_became_active":"2024-03-31T15:55:17.567296+0000","last_became_peered":"2024-03-31T15:55:17.567296+0000","last_unstale":"2024-03-31T15:55:19.550455+0000","last_undegraded":"2024-03-31T15:55:19.550455+0000","last_fullsized":"2024-03-31T15:55:19.550455+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:24:18.305487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554616+0000","last_change":"2024-03-31T15:55:17.569573+0000","last_active":"2024-03-31T15:55:18.554616+0000","last_peered":"2024-03-31T15:55:18.554616+0000","last_clean":"2024-03-31T15:55:18.554616+0000","last_became_active":"2024-03-31T15:55:17.569295+0000","last_became_peered":"2024-03-31T15:55:17.569295+0000","last_unstale":"2024-03-31T15:55:18.554616+0000","last_undegraded":"2024-03-31T15:55:18.554616+0000","last_fullsized":"2024-03-31T15:55:18.554616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:11.923571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548574+0000","last_change":"2024-03-31T15:55:18.581151+0000","last_active":"2024-03-31T15:55:19.548574+0000","last_peered":"2024-03-31T15:55:19.548574+0000","last_clean":"2024-03-31T15:55:19.548574+0000","last_became_active":"2024-03-31T15:55:18.580843+0000","last_became_peered":"2024-03-31T15:55:18.580843+0000","last_unstale":"2024-03-31T15:55:19.548574+0000","last_undegraded":"2024-03-31T15:55:19.548574+0000","last_fullsized":"2024-03-31T15:55:19.548574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550481+0000","last_change":"2024-03-31T15:55:17.561569+0000","last_active":"2024-03-31T15:55:19.550481+0000","last_peered":"2024-03-31T15:55:19.550481+0000","last_clean":"2024-03-31T15:55:19.550481+0000","last_became_active":"2024-03-31T15:55:17.560629+0000","last_became_peered":"2024-03-31T15:55:17.560629+0000","last_unstale":"2024-03-31T15:55:19.550481+0000","last_undegraded":"2024-03-31T15:55:19.550481+0000","last_fullsized":"2024-03-31T15:55:19.550481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:36.826694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583276+0000","last_change":"2024-03-31T15:55:18.576406+0000","last_active":"2024-03-31T15:55:18.583276+0000","last_peered":"2024-03-31T15:55:18.583276+0000","last_clean":"2024-03-31T15:55:18.583276+0000","last_became_active":"2024-03-31T15:55:18.575892+0000","last_became_peered":"2024-03-31T15:55:18.575892+0000","last_unstale":"2024-03-31T15:55:18.583276+0000","last_undegraded":"2024-03-31T15:55:18.583276+0000","last_fullsized":"2024-03-31T15:55:18.583276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:01.439448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.552589+0000","last_change":"2024-03-31T15:55:17.565054+0000","last_active":"2024-03-31T15:55:19.552589+0000","last_peered":"2024-03-31T15:55:19.552589+0000","last_clean":"2024-03-31T15:55:19.552589+0000","last_became_active":"2024-03-31T15:55:17.564845+0000","last_became_peered":"2024-03-31T15:55:17.564845+0000","last_unstale":"2024-03-31T15:55:19.552589+0000","last_undegraded":"2024-03-31T15:55:19.552589+0000","last_fullsized":"2024-03-31T15:55:19.552589+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:34:03.576042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548610+0000","last_change":"2024-03-31T15:55:18.572781+0000","last_active":"2024-03-31T15:55:19.548610+0000","last_peered":"2024-03-31T15:55:19.548610+0000","last_clean":"2024-03-31T15:55:19.548610+0000","last_became_active":"2024-03-31T15:55:18.572475+0000","last_became_peered":"2024-03-31T15:55:18.572475+0000","last_unstale":"2024-03-31T15:55:19.548610+0000","last_undegraded":"2024-03-31T15:55:19.548610+0000","last_fullsized":"2024-03-31T15:55:19.548610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:21.840147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583180+0000","last_change":"2024-03-31T15:55:18.575859+0000","last_active":"2024-03-31T15:55:18.583180+0000","last_peered":"2024-03-31T15:55:18.583180+0000","last_clean":"2024-03-31T15:55:18.583180+0000","last_became_active":"2024-03-31T15:55:18.575507+0000","last_became_peered":"2024-03-31T15:55:18.575507+0000","last_unstale":"2024-03-31T15:55:18.583180+0000","last_undegraded":"2024-03-31T15:55:18.583180+0000","last_fullsized":"2024-03-31T15:55:18.583180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:31:28.813601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547913+0000","last_change":"2024-03-31T15:55:17.559862+0000","last_active":"2024-03-31T15:55:19.547913+0000","last_peered":"2024-03-31T15:55:19.547913+0000","last_clean":"2024-03-31T15:55:19.547913+0000","last_became_active":"2024-03-31T15:55:17.558616+0000","last_became_peered":"2024-03-31T15:55:17.558616+0000","last_unstale":"2024-03-31T15:55:19.547913+0000","last_undegraded":"2024-03-31T15:55:19.547913+0000","last_fullsized":"2024-03-31T15:55:19.547913+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:28.724128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554197+0000","last_change":"2024-03-31T15:55:17.570487+0000","last_active":"2024-03-31T15:55:18.554197+0000","last_peered":"2024-03-31T15:55:18.554197+0000","last_clean":"2024-03-31T15:55:18.554197+0000","last_became_active":"2024-03-31T15:55:17.570204+0000","last_became_peered":"2024-03-31T15:55:17.570204+0000","last_unstale":"2024-03-31T15:55:18.554197+0000","last_undegraded":"2024-03-31T15:55:18.554197+0000","last_fullsized":"2024-03-31T15:55:18.554197+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:34.516949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.553800+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:55:18.553800+0000","last_peered":"2024-03-31T15:55:18.553800+0000","last_clean":"2024-03-31T15:55:18.553800+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:55:18.553800+0000","last_undegraded":"2024-03-31T15:55:18.553800+0000","last_fullsized":"2024-03-31T15:55:18.553800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":52,"acting":52,"num_store_stats":3},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":26,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":755564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":186885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":185309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":777289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:55:22.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:55:22.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.816+0000 7fe7e4000640 1 Processor -- start 2024-03-31T15:55:22.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.816+0000 7fe7e4000640 1 -- start start 2024-03-31T15:55:22.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e4000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7dc15c120 0x7fe7dc15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e4000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7dc0d2b10 con 0x7fe7dc0d2680 2024-03-31T15:55:22.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e4000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7dc160350 con 0x7fe7dc0cca30 2024-03-31T15:55:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e4000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7dc160bf0 con 0x7fe7dc15c120 2024-03-31T15:55:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e0c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe7dc0d2680 0x7fe7dc0b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:52352/0 (socket says 172.21.15.90:52352) 2024-03-31T15:55:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e0c00640 1 -- 172.21.15.90:0/4147815151 learned_addr learned my addr 172.21.15.90:0/4147815151 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:22.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e1600640 1 --2- 172.21.15.90:0/4147815151 >> v2:172.21.15.102:3300/0 conn(0x7fe7dc15c120 0x7fe7dc15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1125662889 0 0) 0x7fe7dc0d2b10 con 0x7fe7dc0d2680 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7c4003530 con 0x7fe7dc0d2680 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 198287136 0 0) 0x7fe7c4003530 con 0x7fe7dc0d2680 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e1600640 1 -- 172.21.15.90:0/4147815151 >> v1:172.21.15.102:6789/0 conn(0x7fe7dc0cca30 legacy=0x7fe7dc0d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e1600640 1 -- 172.21.15.90:0/4147815151 >> v1:172.21.15.90:6789/0 conn(0x7fe7dc0d2680 legacy=0x7fe7dc0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.820+0000 7fe7e1600640 1 -- 172.21.15.90:0/4147815151 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc160630 con 0x7fe7dc15c120 2024-03-31T15:55:22.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e1600640 1 --2- 172.21.15.90:0/4147815151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc15fd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00097c0 tx=0x7fe7d001c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0ba3fa4246ab7ff server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00b0070 con 0x7fe7dc15c120 2024-03-31T15:55:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d00ab400 con 0x7fe7dc15c120 2024-03-31T15:55:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7dbe00640 1 -- 172.21.15.90:0/4147815151 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8001680 con 0x7fe7dc15c120 2024-03-31T15:55:22.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- 172.21.15.90:0/4147815151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 msgr2=0x7fe7dc15fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 --2- 172.21.15.90:0/4147815151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc15fd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00097c0 tx=0x7fe7d001c300 comp rx=0 tx=0).stop 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- 172.21.15.90:0/4147815151 shutdown_connections 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 --2- 172.21.15.90:0/4147815151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc15fd00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- 172.21.15.90:0/4147815151 >> 172.21.15.90:0/4147815151 conn(0x7fe7dc09aa40 msgr2=0x7fe7dc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- 172.21.15.90:0/4147815151 shutdown_connections 2024-03-31T15:55:22.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- 172.21.15.90:0/4147815151 wait complete. 2024-03-31T15:55:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 Processor -- start 2024-03-31T15:55:22.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.824+0000 7fe7e4000640 1 -- start start 2024-03-31T15:55:22.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0d2680 0x7fe7dc15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc0b2f70 con 0x7fe7dc0cca30 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc0b2df0 con 0x7fe7dc15c120 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe7dc0d0bf0 con 0x7fe7dc0d2680 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56682/0 (socket says 172.21.15.90:56682) 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 -- 172.21.15.90:0/2716860709 learned_addr learned my addr 172.21.15.90:0/2716860709 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e0c00640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0d2680 0x7fe7dc15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e2000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 -- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0d2680 msgr2=0x7fe7dc15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0d2680 0x7fe7dc15abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 -- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 msgr2=0x7fe7dc153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 -- 172.21.15.90:0/2716860709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc167e50 con 0x7fe7dc0cca30 2024-03-31T15:55:22.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e2000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e1600640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00ae040 tx=0x7fe7d001ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00bb020 con 0x7fe7dc0cca30 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d00b1040 con 0x7fe7dc0cca30 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d00b0050 con 0x7fe7dc0cca30 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7dc15b590 con 0x7fe7dc0cca30 2024-03-31T15:55:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7dc0aef60 con 0x7fe7dc0cca30 2024-03-31T15:55:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.828+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7dc0686d0 con 0x7fe7dc0cca30 2024-03-31T15:55:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.832+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe7d00be070 con 0x7fe7dc0cca30 2024-03-31T15:55:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.832+0000 7fe7d9600640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 0x7fe7bc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.832+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe7d0120080 con 0x7fe7dc0cca30 2024-03-31T15:55:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.832+0000 7fe7e0c00640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 0x7fe7bc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:22.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.832+0000 7fe7e0c00640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 0x7fe7bc06a870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc1fa3e0 tx=0x7fe7c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:22.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:22.836+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe7d0015fb0 con 0x7fe7dc0cca30 2024-03-31T15:55:23.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.040+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe7dc0b1b60 con 0x7fe7bc0683b0 2024-03-31T15:55:23.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.044+0000 7fe7d9600640 1 -- 172.21.15.90:0/2716860709 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243097 (secure 0 0 0) 0x7fe7dc0b1b60 con 0x7fe7bc0683b0 2024-03-31T15:55:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:23.043 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:55:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 msgr2=0x7fe7bc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 0x7fe7bc06a870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc1fa3e0 tx=0x7fe7c8009040 comp rx=0 tx=0).stop 2024-03-31T15:55:23.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 msgr2=0x7fe7dc167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00ae040 tx=0x7fe7d001ab40 comp rx=0 tx=0).stop 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 shutdown_connections 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe7bc0683b0 0x7fe7bc06a870 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7dc15c120 0x7fe7dc153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe7dc0d2680 0x7fe7dc15abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 --2- 172.21.15.90:0/2716860709 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe7dc0cca30 0x7fe7dc167910 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 >> 172.21.15.90:0/2716860709 conn(0x7fe7dc09aa40 msgr2=0x7fe7dc06f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 shutdown_connections 2024-03-31T15:55:23.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.048+0000 7fe7e4000640 1 -- 172.21.15.90:0/2716860709 wait complete. 2024-03-31T15:55:23.072 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-03-31T15:55:21.238712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592084,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":183,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":80,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":220572,"kb_used_data":4376,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754148,"statfs":{"total":773094113280,"available":772868247552,"internally_reserved":0,"allocated":4481024,"data_stored":2565831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.704321"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548723+0000","last_change":"2024-03-31T15:55:18.569836+0000","last_active":"2024-03-31T15:55:19.548723+0000","last_peered":"2024-03-31T15:55:19.548723+0000","last_clean":"2024-03-31T15:55:19.548723+0000","last_became_active":"2024-03-31T15:55:18.569542+0000","last_became_peered":"2024-03-31T15:55:18.569542+0000","last_unstale":"2024-03-31T15:55:19.548723+0000","last_undegraded":"2024-03-31T15:55:19.548723+0000","last_fullsized":"2024-03-31T15:55:19.548723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:52.818181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548259+0000","last_change":"2024-03-31T15:55:18.570636+0000","last_active":"2024-03-31T15:55:19.548259+0000","last_peered":"2024-03-31T15:55:19.548259+0000","last_clean":"2024-03-31T15:55:19.548259+0000","last_became_active":"2024-03-31T15:55:18.570347+0000","last_became_peered":"2024-03-31T15:55:18.570347+0000","last_unstale":"2024-03-31T15:55:19.548259+0000","last_undegraded":"2024-03-31T15:55:19.548259+0000","last_fullsized":"2024-03-31T15:55:19.548259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550774+0000","last_change":"2024-03-31T15:55:18.558889+0000","last_active":"2024-03-31T15:55:19.550774+0000","last_peered":"2024-03-31T15:55:19.550774+0000","last_clean":"2024-03-31T15:55:19.550774+0000","last_became_active":"2024-03-31T15:55:18.558620+0000","last_became_peered":"2024-03-31T15:55:18.558620+0000","last_unstale":"2024-03-31T15:55:19.550774+0000","last_undegraded":"2024-03-31T15:55:19.550774+0000","last_fullsized":"2024-03-31T15:55:19.550774+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:16.723939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550456+0000","last_change":"2024-03-31T15:55:18.552183+0000","last_active":"2024-03-31T15:55:19.550456+0000","last_peered":"2024-03-31T15:55:19.550456+0000","last_clean":"2024-03-31T15:55:19.550456+0000","last_became_active":"2024-03-31T15:55:18.551886+0000","last_became_peered":"2024-03-31T15:55:18.551886+0000","last_unstale":"2024-03-31T15:55:19.550456+0000","last_undegraded":"2024-03-31T15:55:19.550456+0000","last_fullsized":"2024-03-31T15:55:19.550456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:53:07.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547884+0000","last_change":"2024-03-31T15:55:18.572173+0000","last_active":"2024-03-31T15:55:19.547884+0000","last_peered":"2024-03-31T15:55:19.547884+0000","last_clean":"2024-03-31T15:55:19.547884+0000","last_became_active":"2024-03-31T15:55:18.571909+0000","last_became_peered":"2024-03-31T15:55:18.571909+0000","last_unstale":"2024-03-31T15:55:19.547884+0000","last_undegraded":"2024-03-31T15:55:19.547884+0000","last_fullsized":"2024-03-31T15:55:19.547884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580155+0000","last_change":"2024-03-31T15:55:18.573440+0000","last_active":"2024-03-31T15:55:18.580155+0000","last_peered":"2024-03-31T15:55:18.580155+0000","last_clean":"2024-03-31T15:55:18.580155+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:18.580155+0000","last_undegraded":"2024-03-31T15:55:18.580155+0000","last_fullsized":"2024-03-31T15:55:18.580155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:44.338598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.579763+0000","last_change":"2024-03-31T15:55:18.571940+0000","last_active":"2024-03-31T15:55:18.579763+0000","last_peered":"2024-03-31T15:55:18.579763+0000","last_clean":"2024-03-31T15:55:18.579763+0000","last_became_active":"2024-03-31T15:55:18.571687+0000","last_became_peered":"2024-03-31T15:55:18.571687+0000","last_unstale":"2024-03-31T15:55:18.579763+0000","last_undegraded":"2024-03-31T15:55:18.579763+0000","last_fullsized":"2024-03-31T15:55:18.579763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:24.106111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550838+0000","last_change":"2024-03-31T15:55:18.565501+0000","last_active":"2024-03-31T15:55:19.550838+0000","last_peered":"2024-03-31T15:55:19.550838+0000","last_clean":"2024-03-31T15:55:19.550838+0000","last_became_active":"2024-03-31T15:55:18.565247+0000","last_became_peered":"2024-03-31T15:55:18.565247+0000","last_unstale":"2024-03-31T15:55:19.550838+0000","last_undegraded":"2024-03-31T15:55:19.550838+0000","last_fullsized":"2024-03-31T15:55:19.550838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:55.707922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583121+0000","last_change":"2024-03-31T15:55:18.575290+0000","last_active":"2024-03-31T15:55:18.583121+0000","last_peered":"2024-03-31T15:55:18.583121+0000","last_clean":"2024-03-31T15:55:18.583121+0000","last_became_active":"2024-03-31T15:55:18.574928+0000","last_became_peered":"2024-03-31T15:55:18.574928+0000","last_unstale":"2024-03-31T15:55:18.583121+0000","last_undegraded":"2024-03-31T15:55:18.583121+0000","last_fullsized":"2024-03-31T15:55:18.583121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:04.640612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550628+0000","last_change":"2024-03-31T15:55:18.563439+0000","last_active":"2024-03-31T15:55:19.550628+0000","last_peered":"2024-03-31T15:55:19.550628+0000","last_clean":"2024-03-31T15:55:19.550628+0000","last_became_active":"2024-03-31T15:55:18.563160+0000","last_became_peered":"2024-03-31T15:55:18.563160+0000","last_unstale":"2024-03-31T15:55:19.550628+0000","last_undegraded":"2024-03-31T15:55:19.550628+0000","last_fullsized":"2024-03-31T15:55:19.550628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:33.025431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583094+0000","last_change":"2024-03-31T15:55:18.574506+0000","last_active":"2024-03-31T15:55:18.583094+0000","last_peered":"2024-03-31T15:55:18.583094+0000","last_clean":"2024-03-31T15:55:18.583094+0000","last_became_active":"2024-03-31T15:55:18.570090+0000","last_became_peered":"2024-03-31T15:55:18.570090+0000","last_unstale":"2024-03-31T15:55:18.583094+0000","last_undegraded":"2024-03-31T15:55:18.583094+0000","last_fullsized":"2024-03-31T15:55:18.583094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:14.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580312+0000","last_change":"2024-03-31T15:55:18.570263+0000","last_active":"2024-03-31T15:55:18.580312+0000","last_peered":"2024-03-31T15:55:18.580312+0000","last_clean":"2024-03-31T15:55:18.580312+0000","last_became_active":"2024-03-31T15:55:18.569923+0000","last_became_peered":"2024-03-31T15:55:18.569923+0000","last_unstale":"2024-03-31T15:55:18.580312+0000","last_undegraded":"2024-03-31T15:55:18.580312+0000","last_fullsized":"2024-03-31T15:55:18.580312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:29.731970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548666+0000","last_change":"2024-03-31T15:55:18.583489+0000","last_active":"2024-03-31T15:55:19.548666+0000","last_peered":"2024-03-31T15:55:19.548666+0000","last_clean":"2024-03-31T15:55:19.548666+0000","last_became_active":"2024-03-31T15:55:18.583246+0000","last_became_peered":"2024-03-31T15:55:18.583246+0000","last_unstale":"2024-03-31T15:55:19.548666+0000","last_undegraded":"2024-03-31T15:55:19.548666+0000","last_fullsized":"2024-03-31T15:55:19.548666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:48:46.491057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.577287+0000","last_change":"2024-03-31T15:55:17.569775+0000","last_active":"2024-03-31T15:55:18.577287+0000","last_peered":"2024-03-31T15:55:18.577287+0000","last_clean":"2024-03-31T15:55:18.577287+0000","last_became_active":"2024-03-31T15:55:17.568127+0000","last_became_peered":"2024-03-31T15:55:17.568127+0000","last_unstale":"2024-03-31T15:55:18.577287+0000","last_undegraded":"2024-03-31T15:55:18.577287+0000","last_fullsized":"2024-03-31T15:55:18.577287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:19:15.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548413+0000","last_change":"2024-03-31T15:55:18.572911+0000","last_active":"2024-03-31T15:55:19.548413+0000","last_peered":"2024-03-31T15:55:19.548413+0000","last_clean":"2024-03-31T15:55:19.548413+0000","last_became_active":"2024-03-31T15:55:18.572663+0000","last_became_peered":"2024-03-31T15:55:18.572663+0000","last_unstale":"2024-03-31T15:55:19.548413+0000","last_undegraded":"2024-03-31T15:55:19.548413+0000","last_fullsized":"2024-03-31T15:55:19.548413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550725+0000","last_change":"2024-03-31T15:55:18.558312+0000","last_active":"2024-03-31T15:55:19.550725+0000","last_peered":"2024-03-31T15:55:19.550725+0000","last_clean":"2024-03-31T15:55:19.550725+0000","last_became_active":"2024-03-31T15:55:18.556133+0000","last_became_peered":"2024-03-31T15:55:18.556133+0000","last_unstale":"2024-03-31T15:55:19.550725+0000","last_undegraded":"2024-03-31T15:55:19.550725+0000","last_fullsized":"2024-03-31T15:55:19.550725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:51:25.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548747+0000","last_change":"2024-03-31T15:55:17.559251+0000","last_active":"2024-03-31T15:55:19.548747+0000","last_peered":"2024-03-31T15:55:19.548747+0000","last_clean":"2024-03-31T15:55:19.548747+0000","last_became_active":"2024-03-31T15:55:17.558977+0000","last_became_peered":"2024-03-31T15:55:17.558977+0000","last_unstale":"2024-03-31T15:55:19.548747+0000","last_undegraded":"2024-03-31T15:55:19.548747+0000","last_fullsized":"2024-03-31T15:55:19.548747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:23.098269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583255+0000","last_change":"2024-03-31T15:55:18.576163+0000","last_active":"2024-03-31T15:55:18.583255+0000","last_peered":"2024-03-31T15:55:18.583255+0000","last_clean":"2024-03-31T15:55:18.583255+0000","last_became_active":"2024-03-31T15:55:18.575491+0000","last_became_peered":"2024-03-31T15:55:18.575491+0000","last_unstale":"2024-03-31T15:55:18.583255+0000","last_undegraded":"2024-03-31T15:55:18.583255+0000","last_fullsized":"2024-03-31T15:55:18.583255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:25.287778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548239+0000","last_change":"2024-03-31T15:55:17.561703+0000","last_active":"2024-03-31T15:55:19.548239+0000","last_peered":"2024-03-31T15:55:19.548239+0000","last_clean":"2024-03-31T15:55:19.548239+0000","last_became_active":"2024-03-31T15:55:17.561448+0000","last_became_peered":"2024-03-31T15:55:17.561448+0000","last_unstale":"2024-03-31T15:55:19.548239+0000","last_undegraded":"2024-03-31T15:55:19.548239+0000","last_fullsized":"2024-03-31T15:55:19.548239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:38.684847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548566+0000","last_change":"2024-03-31T15:55:17.565178+0000","last_active":"2024-03-31T15:55:19.548566+0000","last_peered":"2024-03-31T15:55:19.548566+0000","last_clean":"2024-03-31T15:55:19.548566+0000","last_became_active":"2024-03-31T15:55:17.564972+0000","last_became_peered":"2024-03-31T15:55:17.564972+0000","last_unstale":"2024-03-31T15:55:19.548566+0000","last_undegraded":"2024-03-31T15:55:19.548566+0000","last_fullsized":"2024-03-31T15:55:19.548566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:43.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548114+0000","last_change":"2024-03-31T15:55:18.576538+0000","last_active":"2024-03-31T15:55:19.548114+0000","last_peered":"2024-03-31T15:55:19.548114+0000","last_clean":"2024-03-31T15:55:19.548114+0000","last_became_active":"2024-03-31T15:55:18.569912+0000","last_became_peered":"2024-03-31T15:55:18.569912+0000","last_unstale":"2024-03-31T15:55:19.548114+0000","last_undegraded":"2024-03-31T15:55:19.548114+0000","last_fullsized":"2024-03-31T15:55:19.548114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580350+0000","last_change":"2024-03-31T15:55:18.567828+0000","last_active":"2024-03-31T15:55:18.580350+0000","last_peered":"2024-03-31T15:55:18.580350+0000","last_clean":"2024-03-31T15:55:18.580350+0000","last_became_active":"2024-03-31T15:55:18.567508+0000","last_became_peered":"2024-03-31T15:55:18.567508+0000","last_unstale":"2024-03-31T15:55:18.580350+0000","last_undegraded":"2024-03-31T15:55:18.580350+0000","last_fullsized":"2024-03-31T15:55:18.580350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:39.692689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548449+0000","last_change":"2024-03-31T15:55:18.570563+0000","last_active":"2024-03-31T15:55:19.548449+0000","last_peered":"2024-03-31T15:55:19.548449+0000","last_clean":"2024-03-31T15:55:19.548449+0000","last_became_active":"2024-03-31T15:55:18.570292+0000","last_became_peered":"2024-03-31T15:55:18.570292+0000","last_unstale":"2024-03-31T15:55:19.548449+0000","last_undegraded":"2024-03-31T15:55:19.548449+0000","last_fullsized":"2024-03-31T15:55:19.548449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.584994+0000","last_change":"2024-03-31T15:55:17.568544+0000","last_active":"2024-03-31T15:55:18.584994+0000","last_peered":"2024-03-31T15:55:18.584994+0000","last_clean":"2024-03-31T15:55:18.584994+0000","last_became_active":"2024-03-31T15:55:17.567942+0000","last_became_peered":"2024-03-31T15:55:17.567942+0000","last_unstale":"2024-03-31T15:55:18.584994+0000","last_undegraded":"2024-03-31T15:55:18.584994+0000","last_fullsized":"2024-03-31T15:55:18.584994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:10.013322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548171+0000","last_change":"2024-03-31T15:55:18.574899+0000","last_active":"2024-03-31T15:55:19.548171+0000","last_peered":"2024-03-31T15:55:19.548171+0000","last_clean":"2024-03-31T15:55:19.548171+0000","last_became_active":"2024-03-31T15:55:18.574347+0000","last_became_peered":"2024-03-31T15:55:18.574347+0000","last_unstale":"2024-03-31T15:55:19.548171+0000","last_undegraded":"2024-03-31T15:55:19.548171+0000","last_fullsized":"2024-03-31T15:55:19.548171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550772+0000","last_change":"2024-03-31T15:55:17.562464+0000","last_active":"2024-03-31T15:55:19.550772+0000","last_peered":"2024-03-31T15:55:19.550772+0000","last_clean":"2024-03-31T15:55:19.550772+0000","last_became_active":"2024-03-31T15:55:17.562192+0000","last_became_peered":"2024-03-31T15:55:17.562192+0000","last_unstale":"2024-03-31T15:55:19.550772+0000","last_undegraded":"2024-03-31T15:55:19.550772+0000","last_fullsized":"2024-03-31T15:55:19.550772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:13.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550931+0000","last_change":"2024-03-31T15:55:18.572667+0000","last_active":"2024-03-31T15:55:19.550931+0000","last_peered":"2024-03-31T15:55:19.550931+0000","last_clean":"2024-03-31T15:55:19.550931+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.550931+0000","last_undegraded":"2024-03-31T15:55:19.550931+0000","last_fullsized":"2024-03-31T15:55:19.550931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:54:13.587576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583198+0000","last_change":"2024-03-31T15:55:18.573739+0000","last_active":"2024-03-31T15:55:18.583198+0000","last_peered":"2024-03-31T15:55:18.583198+0000","last_clean":"2024-03-31T15:55:18.583198+0000","last_became_active":"2024-03-31T15:55:18.573288+0000","last_became_peered":"2024-03-31T15:55:18.573288+0000","last_unstale":"2024-03-31T15:55:18.583198+0000","last_undegraded":"2024-03-31T15:55:18.583198+0000","last_fullsized":"2024-03-31T15:55:18.583198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:27.032794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550590+0000","last_change":"2024-03-31T15:55:17.561760+0000","last_active":"2024-03-31T15:55:19.550590+0000","last_peered":"2024-03-31T15:55:19.550590+0000","last_clean":"2024-03-31T15:55:19.550590+0000","last_became_active":"2024-03-31T15:55:17.561471+0000","last_became_peered":"2024-03-31T15:55:17.561471+0000","last_unstale":"2024-03-31T15:55:19.550590+0000","last_undegraded":"2024-03-31T15:55:19.550590+0000","last_fullsized":"2024-03-31T15:55:19.550590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:31:27.557663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550455+0000","last_change":"2024-03-31T15:55:17.567522+0000","last_active":"2024-03-31T15:55:19.550455+0000","last_peered":"2024-03-31T15:55:19.550455+0000","last_clean":"2024-03-31T15:55:19.550455+0000","last_became_active":"2024-03-31T15:55:17.567296+0000","last_became_peered":"2024-03-31T15:55:17.567296+0000","last_unstale":"2024-03-31T15:55:19.550455+0000","last_undegraded":"2024-03-31T15:55:19.550455+0000","last_fullsized":"2024-03-31T15:55:19.550455+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:24:18.305487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554616+0000","last_change":"2024-03-31T15:55:17.569573+0000","last_active":"2024-03-31T15:55:18.554616+0000","last_peered":"2024-03-31T15:55:18.554616+0000","last_clean":"2024-03-31T15:55:18.554616+0000","last_became_active":"2024-03-31T15:55:17.569295+0000","last_became_peered":"2024-03-31T15:55:17.569295+0000","last_unstale":"2024-03-31T15:55:18.554616+0000","last_undegraded":"2024-03-31T15:55:18.554616+0000","last_fullsized":"2024-03-31T15:55:18.554616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:11.923571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548574+0000","last_change":"2024-03-31T15:55:18.581151+0000","last_active":"2024-03-31T15:55:19.548574+0000","last_peered":"2024-03-31T15:55:19.548574+0000","last_clean":"2024-03-31T15:55:19.548574+0000","last_became_active":"2024-03-31T15:55:18.580843+0000","last_became_peered":"2024-03-31T15:55:18.580843+0000","last_unstale":"2024-03-31T15:55:19.548574+0000","last_undegraded":"2024-03-31T15:55:19.548574+0000","last_fullsized":"2024-03-31T15:55:19.548574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550481+0000","last_change":"2024-03-31T15:55:17.561569+0000","last_active":"2024-03-31T15:55:19.550481+0000","last_peered":"2024-03-31T15:55:19.550481+0000","last_clean":"2024-03-31T15:55:19.550481+0000","last_became_active":"2024-03-31T15:55:17.560629+0000","last_became_peered":"2024-03-31T15:55:17.560629+0000","last_unstale":"2024-03-31T15:55:19.550481+0000","last_undegraded":"2024-03-31T15:55:19.550481+0000","last_fullsized":"2024-03-31T15:55:19.550481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:36.826694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583276+0000","last_change":"2024-03-31T15:55:18.576406+0000","last_active":"2024-03-31T15:55:18.583276+0000","last_peered":"2024-03-31T15:55:18.583276+0000","last_clean":"2024-03-31T15:55:18.583276+0000","last_became_active":"2024-03-31T15:55:18.575892+0000","last_became_peered":"2024-03-31T15:55:18.575892+0000","last_unstale":"2024-03-31T15:55:18.583276+0000","last_undegraded":"2024-03-31T15:55:18.583276+0000","last_fullsized":"2024-03-31T15:55:18.583276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:01.439448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.552589+0000","last_change":"2024-03-31T15:55:17.565054+0000","last_active":"2024-03-31T15:55:19.552589+0000","last_peered":"2024-03-31T15:55:19.552589+0000","last_clean":"2024-03-31T15:55:19.552589+0000","last_became_active":"2024-03-31T15:55:17.564845+0000","last_became_peered":"2024-03-31T15:55:17.564845+0000","last_unstale":"2024-03-31T15:55:19.552589+0000","last_undegraded":"2024-03-31T15:55:19.552589+0000","last_fullsized":"2024-03-31T15:55:19.552589+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:34:03.576042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548610+0000","last_change":"2024-03-31T15:55:18.572781+0000","last_active":"2024-03-31T15:55:19.548610+0000","last_peered":"2024-03-31T15:55:19.548610+0000","last_clean":"2024-03-31T15:55:19.548610+0000","last_became_active":"2024-03-31T15:55:18.572475+0000","last_became_peered":"2024-03-31T15:55:18.572475+0000","last_unstale":"2024-03-31T15:55:19.548610+0000","last_undegraded":"2024-03-31T15:55:19.548610+0000","last_fullsized":"2024-03-31T15:55:19.548610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:21.840147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:16.524660+0000","last_change":"2024-03-31T15:55:16.524660+0000","last_active":"2024-03-31T15:55:16.524660+0000","last_peered":"2024-03-31T15:55:16.524660+0000","last_clean":"2024-03-31T15:55:16.524660+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:16.524660+0000","last_undegraded":"2024-03-31T15:55:16.524660+0000","last_fullsized":"2024-03-31T15:55:16.524660+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583180+0000","last_change":"2024-03-31T15:55:18.575859+0000","last_active":"2024-03-31T15:55:18.583180+0000","last_peered":"2024-03-31T15:55:18.583180+0000","last_clean":"2024-03-31T15:55:18.583180+0000","last_became_active":"2024-03-31T15:55:18.575507+0000","last_became_peered":"2024-03-31T15:55:18.575507+0000","last_unstale":"2024-03-31T15:55:18.583180+0000","last_undegraded":"2024-03-31T15:55:18.583180+0000","last_fullsized":"2024-03-31T15:55:18.583180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:31:28.813601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547913+0000","last_change":"2024-03-31T15:55:17.559862+0000","last_active":"2024-03-31T15:55:19.547913+0000","last_peered":"2024-03-31T15:55:19.547913+0000","last_clean":"2024-03-31T15:55:19.547913+0000","last_became_active":"2024-03-31T15:55:17.558616+0000","last_became_peered":"2024-03-31T15:55:17.558616+0000","last_unstale":"2024-03-31T15:55:19.547913+0000","last_undegraded":"2024-03-31T15:55:19.547913+0000","last_fullsized":"2024-03-31T15:55:19.547913+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:28.724128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:55:17.532502+0000","last_change":"2024-03-31T15:55:17.532502+0000","last_active":"2024-03-31T15:55:17.532502+0000","last_peered":"2024-03-31T15:55:17.532502+0000","last_clean":"2024-03-31T15:55:17.532502+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:55:17.532502+0000","last_undegraded":"2024-03-31T15:55:17.532502+0000","last_fullsized":"2024-03-31T15:55:17.532502+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554197+0000","last_change":"2024-03-31T15:55:17.570487+0000","last_active":"2024-03-31T15:55:18.554197+0000","last_peered":"2024-03-31T15:55:18.554197+0000","last_clean":"2024-03-31T15:55:18.554197+0000","last_became_active":"2024-03-31T15:55:17.570204+0000","last_became_peered":"2024-03-31T15:55:17.570204+0000","last_unstale":"2024-03-31T15:55:18.554197+0000","last_undegraded":"2024-03-31T15:55:18.554197+0000","last_fullsized":"2024-03-31T15:55:18.554197+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:34.516949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.553800+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:55:18.553800+0000","last_peered":"2024-03-31T15:55:18.553800+0000","last_clean":"2024-03-31T15:55:18.553800+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:55:18.553800+0000","last_undegraded":"2024-03-31T15:55:18.553800+0000","last_fullsized":"2024-03-31T15:55:18.553800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":52,"acting":52,"num_store_stats":3},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1716,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":26,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27416,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509464,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":165196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":987136,"data_stored":755564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":186885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":185309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":777289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:55:23.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:55:23.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.256+0000 7fcc0ae00640 1 Processor -- start 2024-03-31T15:55:23.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.256+0000 7fcc0ae00640 1 -- start start 2024-03-31T15:55:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc0ae00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc0415e1b0 0x7fcc04162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc0406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc0ae00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0415f780 con 0x7fcc0406d5f0 2024-03-31T15:55:23.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc0ae00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0406daa0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcc041622b0 con 0x7fcc0406c7b0 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc0415e1b0 0x7fcc04162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc08e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcc0406d5f0 0x7fcc0415e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44298/0 (socket says 172.21.15.90:44298) 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc08e00640 1 -- 172.21.15.90:0/3362371089 learned_addr learned my addr 172.21.15.90:0/3362371089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcc0415e1b0 0x7fcc04162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50844/0 (socket says 172.21.15.90:50844) 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcbfbe00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc0406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 -- 172.21.15.90:0/3362371089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 msgr2=0x7fcc0406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc0406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 -- 172.21.15.90:0/3362371089 >> v1:172.21.15.90:6789/0 conn(0x7fcc0406d5f0 legacy=0x7fcc0415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03e00640 1 -- 172.21.15.90:0/3362371089 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0415f1c0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.260+0000 7fcc03400640 1 -- 172.21.15.90:0/3362371089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1150157233 0 0) 0x7fcc0415f780 con 0x7fcc0406d5f0 2024-03-31T15:55:23.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc03e00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc04162740 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40097c0 tx=0x7fcbf401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1040ba10d1ecce3d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc03400640 1 -- 172.21.15.90:0/3362371089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40b1070 con 0x7fcc0415e1b0 2024-03-31T15:55:23.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc03400640 1 -- 172.21.15.90:0/3362371089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf40a8050 con 0x7fcc0415e1b0 2024-03-31T15:55:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc03400640 1 -- 172.21.15.90:0/3362371089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40ac860 con 0x7fcc0415e1b0 2024-03-31T15:55:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- 172.21.15.90:0/3362371089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 msgr2=0x7fcc04162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc04162740 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40097c0 tx=0x7fcbf401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- 172.21.15.90:0/3362371089 shutdown_connections 2024-03-31T15:55:23.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc0406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/3362371089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc04162740 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- 172.21.15.90:0/3362371089 >> 172.21.15.90:0/3362371089 conn(0x7fcc0409aa40 msgr2=0x7fcc0409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- 172.21.15.90:0/3362371089 shutdown_connections 2024-03-31T15:55:23.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- 172.21.15.90:0/3362371089 wait complete. 2024-03-31T15:55:23.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 Processor -- start 2024-03-31T15:55:23.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.264+0000 7fcc0ae00640 1 -- start start 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc040c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 0x7fcc040c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc040d2940 con 0x7fcc0415e1b0 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcc040d27c0 con 0x7fcc0406d5f0 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcc040d2ac0 con 0x7fcc0406c7b0 2024-03-31T15:55:23.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50860/0 (socket says 172.21.15.90:50860) 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 -- 172.21.15.90:0/817412495 learned_addr learned my addr 172.21.15.90:0/817412495 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc03e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc040c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcbfbe00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 0x7fcc040c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 -- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 msgr2=0x7fcc040c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc040c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 -- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 msgr2=0x7fcc040c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 0x7fcc040c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 -- 172.21.15.90:0/817412495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0415d9f0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc03e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc040c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcbfbe00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 0x7fcc040c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc08e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c7a0 tx=0x7fcbfc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc018070 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc00a640 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc040c73b0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00aab0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc040c7970 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.268+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc040686d0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.272+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcbfc01b070 con 0x7fcc0415e1b0 2024-03-31T15:55:23.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.272+0000 7fcc00c00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 0x7fcbe006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.272+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcbfc086160 con 0x7fcc0415e1b0 2024-03-31T15:55:23.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.272+0000 7fcc03e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 0x7fcbe006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.272+0000 7fcc03e00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 0x7fcbe006a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40ae000 tx=0x7fcbf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.276+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcbfc0564f0 con 0x7fcc0415e1b0 2024-03-31T15:55:23.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.432+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc040b2c40 con 0x7fcbe00683b0 2024-03-31T15:55:23.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.440+0000 7fcc00c00640 1 -- 172.21.15.90:0/817412495 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7fcc040b2c40 con 0x7fcbe00683b0 2024-03-31T15:55:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:23.437 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:55:23.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 msgr2=0x7fcbe006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 0x7fcbe006a870 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40ae000 tx=0x7fcbf4008040 comp rx=0 tx=0).stop 2024-03-31T15:55:23.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 msgr2=0x7fcc0415d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00c7a0 tx=0x7fcbfc00cc60 comp rx=0 tx=0).stop 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 shutdown_connections 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcbe00683b0 0x7fcbe006a870 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc0415e1b0 0x7fcc0415d4b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0406d5f0 0x7fcc040c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 --2- 172.21.15.90:0/817412495 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcc0406c7b0 0x7fcc040c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 >> 172.21.15.90:0/817412495 conn(0x7fcc0409aa40 msgr2=0x7fcc040b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 shutdown_connections 2024-03-31T15:55:23.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.444+0000 7fcc0ae00640 1 -- 172.21.15.90:0/817412495 wait complete. 2024-03-31T15:55:23.456 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-03-31T15:55:23.239082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220816,"kb_used_data":4632,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753904,"statfs":{"total":773094113280,"available":772867997696,"internally_reserved":0,"allocated":4743168,"data_stored":2694236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001120"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548723+0000","last_change":"2024-03-31T15:55:18.569836+0000","last_active":"2024-03-31T15:55:19.548723+0000","last_peered":"2024-03-31T15:55:19.548723+0000","last_clean":"2024-03-31T15:55:19.548723+0000","last_became_active":"2024-03-31T15:55:18.569542+0000","last_became_peered":"2024-03-31T15:55:18.569542+0000","last_unstale":"2024-03-31T15:55:19.548723+0000","last_undegraded":"2024-03-31T15:55:19.548723+0000","last_fullsized":"2024-03-31T15:55:19.548723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:52.818181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548259+0000","last_change":"2024-03-31T15:55:18.570636+0000","last_active":"2024-03-31T15:55:19.548259+0000","last_peered":"2024-03-31T15:55:19.548259+0000","last_clean":"2024-03-31T15:55:19.548259+0000","last_became_active":"2024-03-31T15:55:18.570347+0000","last_became_peered":"2024-03-31T15:55:18.570347+0000","last_unstale":"2024-03-31T15:55:19.548259+0000","last_undegraded":"2024-03-31T15:55:19.548259+0000","last_fullsized":"2024-03-31T15:55:19.548259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838065+0000","last_change":"2024-03-31T15:55:18.576959+0000","last_active":"2024-03-31T15:55:20.838065+0000","last_peered":"2024-03-31T15:55:20.838065+0000","last_clean":"2024-03-31T15:55:20.838065+0000","last_became_active":"2024-03-31T15:55:18.576703+0000","last_became_peered":"2024-03-31T15:55:18.576703+0000","last_unstale":"2024-03-31T15:55:20.838065+0000","last_undegraded":"2024-03-31T15:55:20.838065+0000","last_fullsized":"2024-03-31T15:55:20.838065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550774+0000","last_change":"2024-03-31T15:55:18.558889+0000","last_active":"2024-03-31T15:55:19.550774+0000","last_peered":"2024-03-31T15:55:19.550774+0000","last_clean":"2024-03-31T15:55:19.550774+0000","last_became_active":"2024-03-31T15:55:18.558620+0000","last_became_peered":"2024-03-31T15:55:18.558620+0000","last_unstale":"2024-03-31T15:55:19.550774+0000","last_undegraded":"2024-03-31T15:55:19.550774+0000","last_fullsized":"2024-03-31T15:55:19.550774+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:16.723939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550456+0000","last_change":"2024-03-31T15:55:18.552183+0000","last_active":"2024-03-31T15:55:19.550456+0000","last_peered":"2024-03-31T15:55:19.550456+0000","last_clean":"2024-03-31T15:55:19.550456+0000","last_became_active":"2024-03-31T15:55:18.551886+0000","last_became_peered":"2024-03-31T15:55:18.551886+0000","last_unstale":"2024-03-31T15:55:19.550456+0000","last_undegraded":"2024-03-31T15:55:19.550456+0000","last_fullsized":"2024-03-31T15:55:19.550456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:53:07.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930104+0000","last_change":"2024-03-31T15:55:18.577900+0000","last_active":"2024-03-31T15:55:20.930104+0000","last_peered":"2024-03-31T15:55:20.930104+0000","last_clean":"2024-03-31T15:55:20.930104+0000","last_became_active":"2024-03-31T15:55:18.577638+0000","last_became_peered":"2024-03-31T15:55:18.577638+0000","last_unstale":"2024-03-31T15:55:20.930104+0000","last_undegraded":"2024-03-31T15:55:20.930104+0000","last_fullsized":"2024-03-31T15:55:20.930104+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553456+0000","last_change":"2024-03-31T15:55:18.569983+0000","last_active":"2024-03-31T15:55:19.553456+0000","last_peered":"2024-03-31T15:55:19.553456+0000","last_clean":"2024-03-31T15:55:19.553456+0000","last_became_active":"2024-03-31T15:55:18.569706+0000","last_became_peered":"2024-03-31T15:55:18.569706+0000","last_unstale":"2024-03-31T15:55:19.553456+0000","last_undegraded":"2024-03-31T15:55:19.553456+0000","last_fullsized":"2024-03-31T15:55:19.553456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:17:54.427505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547884+0000","last_change":"2024-03-31T15:55:18.572173+0000","last_active":"2024-03-31T15:55:19.547884+0000","last_peered":"2024-03-31T15:55:19.547884+0000","last_clean":"2024-03-31T15:55:19.547884+0000","last_became_active":"2024-03-31T15:55:18.571909+0000","last_became_peered":"2024-03-31T15:55:18.571909+0000","last_unstale":"2024-03-31T15:55:19.547884+0000","last_undegraded":"2024-03-31T15:55:19.547884+0000","last_fullsized":"2024-03-31T15:55:19.547884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838187+0000","last_change":"2024-03-31T15:55:18.577912+0000","last_active":"2024-03-31T15:55:20.838187+0000","last_peered":"2024-03-31T15:55:20.838187+0000","last_clean":"2024-03-31T15:55:20.838187+0000","last_became_active":"2024-03-31T15:55:18.574214+0000","last_became_peered":"2024-03-31T15:55:18.574214+0000","last_unstale":"2024-03-31T15:55:20.838187+0000","last_undegraded":"2024-03-31T15:55:20.838187+0000","last_fullsized":"2024-03-31T15:55:20.838187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:48.143140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553957+0000","last_change":"2024-03-31T15:55:18.567327+0000","last_active":"2024-03-31T15:55:19.553957+0000","last_peered":"2024-03-31T15:55:19.553957+0000","last_clean":"2024-03-31T15:55:19.553957+0000","last_became_active":"2024-03-31T15:55:18.567046+0000","last_became_peered":"2024-03-31T15:55:18.567046+0000","last_unstale":"2024-03-31T15:55:19.553957+0000","last_undegraded":"2024-03-31T15:55:19.553957+0000","last_fullsized":"2024-03-31T15:55:19.553957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:21.180135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553799+0000","last_change":"2024-03-31T15:55:18.562702+0000","last_active":"2024-03-31T15:55:19.553799+0000","last_peered":"2024-03-31T15:55:19.553799+0000","last_clean":"2024-03-31T15:55:19.553799+0000","last_became_active":"2024-03-31T15:55:18.562447+0000","last_became_peered":"2024-03-31T15:55:18.562447+0000","last_unstale":"2024-03-31T15:55:19.553799+0000","last_undegraded":"2024-03-31T15:55:19.553799+0000","last_fullsized":"2024-03-31T15:55:19.553799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:59:15.558471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838024+0000","last_change":"2024-03-31T15:55:18.562485+0000","last_active":"2024-03-31T15:55:20.838024+0000","last_peered":"2024-03-31T15:55:20.838024+0000","last_clean":"2024-03-31T15:55:20.838024+0000","last_became_active":"2024-03-31T15:55:18.562260+0000","last_became_peered":"2024-03-31T15:55:18.562260+0000","last_unstale":"2024-03-31T15:55:20.838024+0000","last_undegraded":"2024-03-31T15:55:20.838024+0000","last_fullsized":"2024-03-31T15:55:20.838024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553436+0000","last_change":"2024-03-31T15:55:18.564320+0000","last_active":"2024-03-31T15:55:19.553436+0000","last_peered":"2024-03-31T15:55:19.553436+0000","last_clean":"2024-03-31T15:55:19.553436+0000","last_became_active":"2024-03-31T15:55:18.564064+0000","last_became_peered":"2024-03-31T15:55:18.564064+0000","last_unstale":"2024-03-31T15:55:19.553436+0000","last_undegraded":"2024-03-31T15:55:19.553436+0000","last_fullsized":"2024-03-31T15:55:19.553436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:06.438509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120606+0000","last_change":"2024-03-31T15:55:18.581725+0000","last_active":"2024-03-31T15:55:21.120606+0000","last_peered":"2024-03-31T15:55:21.120606+0000","last_clean":"2024-03-31T15:55:21.120606+0000","last_became_active":"2024-03-31T15:55:18.581115+0000","last_became_peered":"2024-03-31T15:55:18.581115+0000","last_unstale":"2024-03-31T15:55:21.120606+0000","last_undegraded":"2024-03-31T15:55:21.120606+0000","last_fullsized":"2024-03-31T15:55:21.120606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553656+0000","last_change":"2024-03-31T15:55:18.562134+0000","last_active":"2024-03-31T15:55:19.553656+0000","last_peered":"2024-03-31T15:55:19.553656+0000","last_clean":"2024-03-31T15:55:19.553656+0000","last_became_active":"2024-03-31T15:55:18.559688+0000","last_became_peered":"2024-03-31T15:55:18.559688+0000","last_unstale":"2024-03-31T15:55:19.553656+0000","last_undegraded":"2024-03-31T15:55:19.553656+0000","last_fullsized":"2024-03-31T15:55:19.553656+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:57.153194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580155+0000","last_change":"2024-03-31T15:55:18.573440+0000","last_active":"2024-03-31T15:55:18.580155+0000","last_peered":"2024-03-31T15:55:18.580155+0000","last_clean":"2024-03-31T15:55:18.580155+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:18.580155+0000","last_undegraded":"2024-03-31T15:55:18.580155+0000","last_fullsized":"2024-03-31T15:55:18.580155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:44.338598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.579763+0000","last_change":"2024-03-31T15:55:18.571940+0000","last_active":"2024-03-31T15:55:18.579763+0000","last_peered":"2024-03-31T15:55:18.579763+0000","last_clean":"2024-03-31T15:55:18.579763+0000","last_became_active":"2024-03-31T15:55:18.571687+0000","last_became_peered":"2024-03-31T15:55:18.571687+0000","last_unstale":"2024-03-31T15:55:18.579763+0000","last_undegraded":"2024-03-31T15:55:18.579763+0000","last_fullsized":"2024-03-31T15:55:18.579763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:24.106111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547503+0000","last_change":"2024-03-31T15:55:18.581100+0000","last_active":"2024-03-31T15:55:19.547503+0000","last_peered":"2024-03-31T15:55:19.547503+0000","last_clean":"2024-03-31T15:55:19.547503+0000","last_became_active":"2024-03-31T15:55:18.580799+0000","last_became_peered":"2024-03-31T15:55:18.580799+0000","last_unstale":"2024-03-31T15:55:19.547503+0000","last_undegraded":"2024-03-31T15:55:19.547503+0000","last_fullsized":"2024-03-31T15:55:19.547503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550838+0000","last_change":"2024-03-31T15:55:18.565501+0000","last_active":"2024-03-31T15:55:19.550838+0000","last_peered":"2024-03-31T15:55:19.550838+0000","last_clean":"2024-03-31T15:55:19.550838+0000","last_became_active":"2024-03-31T15:55:18.565247+0000","last_became_peered":"2024-03-31T15:55:18.565247+0000","last_unstale":"2024-03-31T15:55:19.550838+0000","last_undegraded":"2024-03-31T15:55:19.550838+0000","last_fullsized":"2024-03-31T15:55:19.550838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:55.707922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:18.576219+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:18.575905+0000","last_became_peered":"2024-03-31T15:55:18.575905+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553636+0000","last_change":"2024-03-31T15:55:18.559315+0000","last_active":"2024-03-31T15:55:19.553636+0000","last_peered":"2024-03-31T15:55:19.553636+0000","last_clean":"2024-03-31T15:55:19.553636+0000","last_became_active":"2024-03-31T15:55:18.558870+0000","last_became_peered":"2024-03-31T15:55:18.558870+0000","last_unstale":"2024-03-31T15:55:19.553636+0000","last_undegraded":"2024-03-31T15:55:19.553636+0000","last_fullsized":"2024-03-31T15:55:19.553636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:45.294220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121007+0000","last_change":"2024-03-31T15:55:18.571748+0000","last_active":"2024-03-31T15:55:21.121007+0000","last_peered":"2024-03-31T15:55:21.121007+0000","last_clean":"2024-03-31T15:55:21.121007+0000","last_became_active":"2024-03-31T15:55:18.571445+0000","last_became_peered":"2024-03-31T15:55:18.571445+0000","last_unstale":"2024-03-31T15:55:21.121007+0000","last_undegraded":"2024-03-31T15:55:21.121007+0000","last_fullsized":"2024-03-31T15:55:21.121007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583121+0000","last_change":"2024-03-31T15:55:18.575290+0000","last_active":"2024-03-31T15:55:18.583121+0000","last_peered":"2024-03-31T15:55:18.583121+0000","last_clean":"2024-03-31T15:55:18.583121+0000","last_became_active":"2024-03-31T15:55:18.574928+0000","last_became_peered":"2024-03-31T15:55:18.574928+0000","last_unstale":"2024-03-31T15:55:18.583121+0000","last_undegraded":"2024-03-31T15:55:18.583121+0000","last_fullsized":"2024-03-31T15:55:18.583121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:04.640612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550628+0000","last_change":"2024-03-31T15:55:18.563439+0000","last_active":"2024-03-31T15:55:19.550628+0000","last_peered":"2024-03-31T15:55:19.550628+0000","last_clean":"2024-03-31T15:55:19.550628+0000","last_became_active":"2024-03-31T15:55:18.563160+0000","last_became_peered":"2024-03-31T15:55:18.563160+0000","last_unstale":"2024-03-31T15:55:19.550628+0000","last_undegraded":"2024-03-31T15:55:19.550628+0000","last_fullsized":"2024-03-31T15:55:19.550628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:33.025431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547597+0000","last_change":"2024-03-31T15:55:18.582358+0000","last_active":"2024-03-31T15:55:19.547597+0000","last_peered":"2024-03-31T15:55:19.547597+0000","last_clean":"2024-03-31T15:55:19.547597+0000","last_became_active":"2024-03-31T15:55:18.581841+0000","last_became_peered":"2024-03-31T15:55:18.581841+0000","last_unstale":"2024-03-31T15:55:19.547597+0000","last_undegraded":"2024-03-31T15:55:19.547597+0000","last_fullsized":"2024-03-31T15:55:19.547597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930482+0000","last_change":"2024-03-31T15:55:18.570686+0000","last_active":"2024-03-31T15:55:20.930482+0000","last_peered":"2024-03-31T15:55:20.930482+0000","last_clean":"2024-03-31T15:55:20.930482+0000","last_became_active":"2024-03-31T15:55:18.570424+0000","last_became_peered":"2024-03-31T15:55:18.570424+0000","last_unstale":"2024-03-31T15:55:20.930482+0000","last_undegraded":"2024-03-31T15:55:20.930482+0000","last_fullsized":"2024-03-31T15:55:20.930482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547692+0000","last_change":"2024-03-31T15:55:18.574827+0000","last_active":"2024-03-31T15:55:19.547692+0000","last_peered":"2024-03-31T15:55:19.547692+0000","last_clean":"2024-03-31T15:55:19.547692+0000","last_became_active":"2024-03-31T15:55:18.574539+0000","last_became_peered":"2024-03-31T15:55:18.574539+0000","last_unstale":"2024-03-31T15:55:19.547692+0000","last_undegraded":"2024-03-31T15:55:19.547692+0000","last_fullsized":"2024-03-31T15:55:19.547692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553659+0000","last_change":"2024-03-31T15:55:18.581864+0000","last_active":"2024-03-31T15:55:19.553659+0000","last_peered":"2024-03-31T15:55:19.553659+0000","last_clean":"2024-03-31T15:55:19.553659+0000","last_became_active":"2024-03-31T15:55:18.579446+0000","last_became_peered":"2024-03-31T15:55:18.579446+0000","last_unstale":"2024-03-31T15:55:19.553659+0000","last_undegraded":"2024-03-31T15:55:19.553659+0000","last_fullsized":"2024-03-31T15:55:19.553659+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:21.190346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547421+0000","last_change":"2024-03-31T15:55:18.581228+0000","last_active":"2024-03-31T15:55:19.547421+0000","last_peered":"2024-03-31T15:55:19.547421+0000","last_clean":"2024-03-31T15:55:19.547421+0000","last_became_active":"2024-03-31T15:55:18.580930+0000","last_became_peered":"2024-03-31T15:55:18.580930+0000","last_unstale":"2024-03-31T15:55:19.547421+0000","last_undegraded":"2024-03-31T15:55:19.547421+0000","last_fullsized":"2024-03-31T15:55:19.547421+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:48.538215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547145+0000","last_change":"2024-03-31T15:55:18.576569+0000","last_active":"2024-03-31T15:55:19.547145+0000","last_peered":"2024-03-31T15:55:19.547145+0000","last_clean":"2024-03-31T15:55:19.547145+0000","last_became_active":"2024-03-31T15:55:18.576267+0000","last_became_peered":"2024-03-31T15:55:18.576267+0000","last_unstale":"2024-03-31T15:55:19.547145+0000","last_undegraded":"2024-03-31T15:55:19.547145+0000","last_fullsized":"2024-03-31T15:55:19.547145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:02.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583094+0000","last_change":"2024-03-31T15:55:18.574506+0000","last_active":"2024-03-31T15:55:18.583094+0000","last_peered":"2024-03-31T15:55:18.583094+0000","last_clean":"2024-03-31T15:55:18.583094+0000","last_became_active":"2024-03-31T15:55:18.570090+0000","last_became_peered":"2024-03-31T15:55:18.570090+0000","last_unstale":"2024-03-31T15:55:18.583094+0000","last_undegraded":"2024-03-31T15:55:18.583094+0000","last_fullsized":"2024-03-31T15:55:18.583094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:14.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930154+0000","last_change":"2024-03-31T15:55:18.560243+0000","last_active":"2024-03-31T15:55:20.930154+0000","last_peered":"2024-03-31T15:55:20.930154+0000","last_clean":"2024-03-31T15:55:20.930154+0000","last_became_active":"2024-03-31T15:55:18.559913+0000","last_became_peered":"2024-03-31T15:55:18.559913+0000","last_unstale":"2024-03-31T15:55:20.930154+0000","last_undegraded":"2024-03-31T15:55:20.930154+0000","last_fullsized":"2024-03-31T15:55:20.930154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580312+0000","last_change":"2024-03-31T15:55:18.570263+0000","last_active":"2024-03-31T15:55:18.580312+0000","last_peered":"2024-03-31T15:55:18.580312+0000","last_clean":"2024-03-31T15:55:18.580312+0000","last_became_active":"2024-03-31T15:55:18.569923+0000","last_became_peered":"2024-03-31T15:55:18.569923+0000","last_unstale":"2024-03-31T15:55:18.580312+0000","last_undegraded":"2024-03-31T15:55:18.580312+0000","last_fullsized":"2024-03-31T15:55:18.580312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:29.731970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:17.569534+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:17.568983+0000","last_became_peered":"2024-03-31T15:55:17.568983+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930469+0000","last_change":"2024-03-31T15:55:18.575181+0000","last_active":"2024-03-31T15:55:20.930469+0000","last_peered":"2024-03-31T15:55:20.930469+0000","last_clean":"2024-03-31T15:55:20.930469+0000","last_became_active":"2024-03-31T15:55:18.574878+0000","last_became_peered":"2024-03-31T15:55:18.574878+0000","last_unstale":"2024-03-31T15:55:20.930469+0000","last_undegraded":"2024-03-31T15:55:20.930469+0000","last_fullsized":"2024-03-31T15:55:20.930469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838089+0000","last_change":"2024-03-31T15:55:17.561836+0000","last_active":"2024-03-31T15:55:20.838089+0000","last_peered":"2024-03-31T15:55:20.838089+0000","last_clean":"2024-03-31T15:55:20.838089+0000","last_became_active":"2024-03-31T15:55:17.561561+0000","last_became_peered":"2024-03-31T15:55:17.561561+0000","last_unstale":"2024-03-31T15:55:20.838089+0000","last_undegraded":"2024-03-31T15:55:20.838089+0000","last_fullsized":"2024-03-31T15:55:20.838089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121107+0000","last_change":"2024-03-31T15:55:18.580605+0000","last_active":"2024-03-31T15:55:21.121107+0000","last_peered":"2024-03-31T15:55:21.121107+0000","last_clean":"2024-03-31T15:55:21.121107+0000","last_became_active":"2024-03-31T15:55:18.580334+0000","last_became_peered":"2024-03-31T15:55:18.580334+0000","last_unstale":"2024-03-31T15:55:21.121107+0000","last_undegraded":"2024-03-31T15:55:21.121107+0000","last_fullsized":"2024-03-31T15:55:21.121107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930625+0000","last_change":"2024-03-31T15:55:17.566422+0000","last_active":"2024-03-31T15:55:20.930625+0000","last_peered":"2024-03-31T15:55:20.930625+0000","last_clean":"2024-03-31T15:55:20.930625+0000","last_became_active":"2024-03-31T15:55:17.566154+0000","last_became_peered":"2024-03-31T15:55:17.566154+0000","last_unstale":"2024-03-31T15:55:20.930625+0000","last_undegraded":"2024-03-31T15:55:20.930625+0000","last_fullsized":"2024-03-31T15:55:20.930625+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:42:15.816385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548666+0000","last_change":"2024-03-31T15:55:18.583489+0000","last_active":"2024-03-31T15:55:19.548666+0000","last_peered":"2024-03-31T15:55:19.548666+0000","last_clean":"2024-03-31T15:55:19.548666+0000","last_became_active":"2024-03-31T15:55:18.583246+0000","last_became_peered":"2024-03-31T15:55:18.583246+0000","last_unstale":"2024-03-31T15:55:19.548666+0000","last_undegraded":"2024-03-31T15:55:19.548666+0000","last_fullsized":"2024-03-31T15:55:19.548666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:48:46.491057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120947+0000","last_change":"2024-03-31T15:55:17.563636+0000","last_active":"2024-03-31T15:55:21.120947+0000","last_peered":"2024-03-31T15:55:21.120947+0000","last_clean":"2024-03-31T15:55:21.120947+0000","last_became_active":"2024-03-31T15:55:17.562976+0000","last_became_peered":"2024-03-31T15:55:17.562976+0000","last_unstale":"2024-03-31T15:55:21.120947+0000","last_undegraded":"2024-03-31T15:55:21.120947+0000","last_fullsized":"2024-03-31T15:55:21.120947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837772+0000","last_change":"2024-03-31T15:55:18.567428+0000","last_active":"2024-03-31T15:55:20.837772+0000","last_peered":"2024-03-31T15:55:20.837772+0000","last_clean":"2024-03-31T15:55:20.837772+0000","last_became_active":"2024-03-31T15:55:18.566876+0000","last_became_peered":"2024-03-31T15:55:18.566876+0000","last_unstale":"2024-03-31T15:55:20.837772+0000","last_undegraded":"2024-03-31T15:55:20.837772+0000","last_fullsized":"2024-03-31T15:55:20.837772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553620+0000","last_change":"2024-03-31T15:55:17.564638+0000","last_active":"2024-03-31T15:55:19.553620+0000","last_peered":"2024-03-31T15:55:19.553620+0000","last_clean":"2024-03-31T15:55:19.553620+0000","last_became_active":"2024-03-31T15:55:17.564347+0000","last_became_peered":"2024-03-31T15:55:17.564347+0000","last_unstale":"2024-03-31T15:55:19.553620+0000","last_undegraded":"2024-03-31T15:55:19.553620+0000","last_fullsized":"2024-03-31T15:55:19.553620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:17.020690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553651+0000","last_change":"2024-03-31T15:55:18.567191+0000","last_active":"2024-03-31T15:55:19.553651+0000","last_peered":"2024-03-31T15:55:19.553651+0000","last_clean":"2024-03-31T15:55:19.553651+0000","last_became_active":"2024-03-31T15:55:18.566900+0000","last_became_peered":"2024-03-31T15:55:18.566900+0000","last_unstale":"2024-03-31T15:55:19.553651+0000","last_undegraded":"2024-03-31T15:55:19.553651+0000","last_fullsized":"2024-03-31T15:55:19.553651+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:03.221135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.577287+0000","last_change":"2024-03-31T15:55:17.569775+0000","last_active":"2024-03-31T15:55:18.577287+0000","last_peered":"2024-03-31T15:55:18.577287+0000","last_clean":"2024-03-31T15:55:18.577287+0000","last_became_active":"2024-03-31T15:55:17.568127+0000","last_became_peered":"2024-03-31T15:55:17.568127+0000","last_unstale":"2024-03-31T15:55:18.577287+0000","last_undegraded":"2024-03-31T15:55:18.577287+0000","last_fullsized":"2024-03-31T15:55:18.577287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:19:15.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121038+0000","last_change":"2024-03-31T15:55:18.573156+0000","last_active":"2024-03-31T15:55:21.121038+0000","last_peered":"2024-03-31T15:55:21.121038+0000","last_clean":"2024-03-31T15:55:21.121038+0000","last_became_active":"2024-03-31T15:55:18.572316+0000","last_became_peered":"2024-03-31T15:55:18.572316+0000","last_unstale":"2024-03-31T15:55:21.121038+0000","last_undegraded":"2024-03-31T15:55:21.121038+0000","last_fullsized":"2024-03-31T15:55:21.121038+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:17.564811+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:17.564461+0000","last_became_peered":"2024-03-31T15:55:17.564461+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548413+0000","last_change":"2024-03-31T15:55:18.572911+0000","last_active":"2024-03-31T15:55:19.548413+0000","last_peered":"2024-03-31T15:55:19.548413+0000","last_clean":"2024-03-31T15:55:19.548413+0000","last_became_active":"2024-03-31T15:55:18.572663+0000","last_became_peered":"2024-03-31T15:55:18.572663+0000","last_unstale":"2024-03-31T15:55:19.548413+0000","last_undegraded":"2024-03-31T15:55:19.548413+0000","last_fullsized":"2024-03-31T15:55:19.548413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120482+0000","last_change":"2024-03-31T15:55:17.558598+0000","last_active":"2024-03-31T15:55:21.120482+0000","last_peered":"2024-03-31T15:55:21.120482+0000","last_clean":"2024-03-31T15:55:21.120482+0000","last_became_active":"2024-03-31T15:55:17.557967+0000","last_became_peered":"2024-03-31T15:55:17.557967+0000","last_unstale":"2024-03-31T15:55:21.120482+0000","last_undegraded":"2024-03-31T15:55:21.120482+0000","last_fullsized":"2024-03-31T15:55:21.120482+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550725+0000","last_change":"2024-03-31T15:55:18.558312+0000","last_active":"2024-03-31T15:55:19.550725+0000","last_peered":"2024-03-31T15:55:19.550725+0000","last_clean":"2024-03-31T15:55:19.550725+0000","last_became_active":"2024-03-31T15:55:18.556133+0000","last_became_peered":"2024-03-31T15:55:18.556133+0000","last_unstale":"2024-03-31T15:55:19.550725+0000","last_undegraded":"2024-03-31T15:55:19.550725+0000","last_fullsized":"2024-03-31T15:55:19.550725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:51:25.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548747+0000","last_change":"2024-03-31T15:55:17.559251+0000","last_active":"2024-03-31T15:55:19.548747+0000","last_peered":"2024-03-31T15:55:19.548747+0000","last_clean":"2024-03-31T15:55:19.548747+0000","last_became_active":"2024-03-31T15:55:17.558977+0000","last_became_peered":"2024-03-31T15:55:17.558977+0000","last_unstale":"2024-03-31T15:55:19.548747+0000","last_undegraded":"2024-03-31T15:55:19.548747+0000","last_fullsized":"2024-03-31T15:55:19.548747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:23.098269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583255+0000","last_change":"2024-03-31T15:55:18.576163+0000","last_active":"2024-03-31T15:55:18.583255+0000","last_peered":"2024-03-31T15:55:18.583255+0000","last_clean":"2024-03-31T15:55:18.583255+0000","last_became_active":"2024-03-31T15:55:18.575491+0000","last_became_peered":"2024-03-31T15:55:18.575491+0000","last_unstale":"2024-03-31T15:55:18.583255+0000","last_undegraded":"2024-03-31T15:55:18.583255+0000","last_fullsized":"2024-03-31T15:55:18.583255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:25.287778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553997+0000","last_change":"2024-03-31T15:55:17.559025+0000","last_active":"2024-03-31T15:55:19.553997+0000","last_peered":"2024-03-31T15:55:19.553997+0000","last_clean":"2024-03-31T15:55:19.553997+0000","last_became_active":"2024-03-31T15:55:17.558732+0000","last_became_peered":"2024-03-31T15:55:17.558732+0000","last_unstale":"2024-03-31T15:55:19.553997+0000","last_undegraded":"2024-03-31T15:55:19.553997+0000","last_fullsized":"2024-03-31T15:55:19.553997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:03.959563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838073+0000","last_change":"2024-03-31T15:55:18.559732+0000","last_active":"2024-03-31T15:55:20.838073+0000","last_peered":"2024-03-31T15:55:20.838073+0000","last_clean":"2024-03-31T15:55:20.838073+0000","last_became_active":"2024-03-31T15:55:18.559432+0000","last_became_peered":"2024-03-31T15:55:18.559432+0000","last_unstale":"2024-03-31T15:55:20.838073+0000","last_undegraded":"2024-03-31T15:55:20.838073+0000","last_fullsized":"2024-03-31T15:55:20.838073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:15.200810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:17.559993+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:17.559669+0000","last_became_peered":"2024-03-31T15:55:17.559669+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:24:33.510910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121078+0000","last_change":"2024-03-31T15:55:18.558993+0000","last_active":"2024-03-31T15:55:21.121078+0000","last_peered":"2024-03-31T15:55:21.121078+0000","last_clean":"2024-03-31T15:55:21.121078+0000","last_became_active":"2024-03-31T15:55:18.558670+0000","last_became_peered":"2024-03-31T15:55:18.558670+0000","last_unstale":"2024-03-31T15:55:21.121078+0000","last_undegraded":"2024-03-31T15:55:21.121078+0000","last_fullsized":"2024-03-31T15:55:21.121078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548239+0000","last_change":"2024-03-31T15:55:17.561703+0000","last_active":"2024-03-31T15:55:19.548239+0000","last_peered":"2024-03-31T15:55:19.548239+0000","last_clean":"2024-03-31T15:55:19.548239+0000","last_became_active":"2024-03-31T15:55:17.561448+0000","last_became_peered":"2024-03-31T15:55:17.561448+0000","last_unstale":"2024-03-31T15:55:19.548239+0000","last_undegraded":"2024-03-31T15:55:19.548239+0000","last_fullsized":"2024-03-31T15:55:19.548239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:38.684847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837750+0000","last_change":"2024-03-31T15:55:18.573704+0000","last_active":"2024-03-31T15:55:20.837750+0000","last_peered":"2024-03-31T15:55:20.837750+0000","last_clean":"2024-03-31T15:55:20.837750+0000","last_became_active":"2024-03-31T15:55:18.573462+0000","last_became_peered":"2024-03-31T15:55:18.573462+0000","last_unstale":"2024-03-31T15:55:20.837750+0000","last_undegraded":"2024-03-31T15:55:20.837750+0000","last_fullsized":"2024-03-31T15:55:20.837750+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548566+0000","last_change":"2024-03-31T15:55:17.565178+0000","last_active":"2024-03-31T15:55:19.548566+0000","last_peered":"2024-03-31T15:55:19.548566+0000","last_clean":"2024-03-31T15:55:19.548566+0000","last_became_active":"2024-03-31T15:55:17.564972+0000","last_became_peered":"2024-03-31T15:55:17.564972+0000","last_unstale":"2024-03-31T15:55:19.548566+0000","last_undegraded":"2024-03-31T15:55:19.548566+0000","last_fullsized":"2024-03-31T15:55:19.548566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:43.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548114+0000","last_change":"2024-03-31T15:55:18.576538+0000","last_active":"2024-03-31T15:55:19.548114+0000","last_peered":"2024-03-31T15:55:19.548114+0000","last_clean":"2024-03-31T15:55:19.548114+0000","last_became_active":"2024-03-31T15:55:18.569912+0000","last_became_peered":"2024-03-31T15:55:18.569912+0000","last_unstale":"2024-03-31T15:55:19.548114+0000","last_undegraded":"2024-03-31T15:55:19.548114+0000","last_fullsized":"2024-03-31T15:55:19.548114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930236+0000","last_change":"2024-03-31T15:55:17.564498+0000","last_active":"2024-03-31T15:55:20.930236+0000","last_peered":"2024-03-31T15:55:20.930236+0000","last_clean":"2024-03-31T15:55:20.930236+0000","last_became_active":"2024-03-31T15:55:17.564148+0000","last_became_peered":"2024-03-31T15:55:17.564148+0000","last_unstale":"2024-03-31T15:55:20.930236+0000","last_undegraded":"2024-03-31T15:55:20.930236+0000","last_fullsized":"2024-03-31T15:55:20.930236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:42:11.639287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580350+0000","last_change":"2024-03-31T15:55:18.567828+0000","last_active":"2024-03-31T15:55:18.580350+0000","last_peered":"2024-03-31T15:55:18.580350+0000","last_clean":"2024-03-31T15:55:18.580350+0000","last_became_active":"2024-03-31T15:55:18.567508+0000","last_became_peered":"2024-03-31T15:55:18.567508+0000","last_unstale":"2024-03-31T15:55:18.580350+0000","last_undegraded":"2024-03-31T15:55:18.580350+0000","last_fullsized":"2024-03-31T15:55:18.580350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:39.692689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547389+0000","last_change":"2024-03-31T15:55:17.568236+0000","last_active":"2024-03-31T15:55:19.547389+0000","last_peered":"2024-03-31T15:55:19.547389+0000","last_clean":"2024-03-31T15:55:19.547389+0000","last_became_active":"2024-03-31T15:55:17.567952+0000","last_became_peered":"2024-03-31T15:55:17.567952+0000","last_unstale":"2024-03-31T15:55:19.547389+0000","last_undegraded":"2024-03-31T15:55:19.547389+0000","last_fullsized":"2024-03-31T15:55:19.547389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548449+0000","last_change":"2024-03-31T15:55:18.570563+0000","last_active":"2024-03-31T15:55:19.548449+0000","last_peered":"2024-03-31T15:55:19.548449+0000","last_clean":"2024-03-31T15:55:19.548449+0000","last_became_active":"2024-03-31T15:55:18.570292+0000","last_became_peered":"2024-03-31T15:55:18.570292+0000","last_unstale":"2024-03-31T15:55:19.548449+0000","last_undegraded":"2024-03-31T15:55:19.548449+0000","last_fullsized":"2024-03-31T15:55:19.548449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.584994+0000","last_change":"2024-03-31T15:55:17.568544+0000","last_active":"2024-03-31T15:55:18.584994+0000","last_peered":"2024-03-31T15:55:18.584994+0000","last_clean":"2024-03-31T15:55:18.584994+0000","last_became_active":"2024-03-31T15:55:17.567942+0000","last_became_peered":"2024-03-31T15:55:17.567942+0000","last_unstale":"2024-03-31T15:55:18.584994+0000","last_undegraded":"2024-03-31T15:55:18.584994+0000","last_fullsized":"2024-03-31T15:55:18.584994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:10.013322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548171+0000","last_change":"2024-03-31T15:55:18.574899+0000","last_active":"2024-03-31T15:55:19.548171+0000","last_peered":"2024-03-31T15:55:19.548171+0000","last_clean":"2024-03-31T15:55:19.548171+0000","last_became_active":"2024-03-31T15:55:18.574347+0000","last_became_peered":"2024-03-31T15:55:18.574347+0000","last_unstale":"2024-03-31T15:55:19.548171+0000","last_undegraded":"2024-03-31T15:55:19.548171+0000","last_fullsized":"2024-03-31T15:55:19.548171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554076+0000","last_change":"2024-03-31T15:55:17.564758+0000","last_active":"2024-03-31T15:55:19.554076+0000","last_peered":"2024-03-31T15:55:19.554076+0000","last_clean":"2024-03-31T15:55:19.554076+0000","last_became_active":"2024-03-31T15:55:17.564417+0000","last_became_peered":"2024-03-31T15:55:17.564417+0000","last_unstale":"2024-03-31T15:55:19.554076+0000","last_undegraded":"2024-03-31T15:55:19.554076+0000","last_fullsized":"2024-03-31T15:55:19.554076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:01.251935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:18.569677+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:18.569325+0000","last_became_peered":"2024-03-31T15:55:18.569325+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:50:35.297846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121071+0000","last_change":"2024-03-31T15:55:17.563289+0000","last_active":"2024-03-31T15:55:21.121071+0000","last_peered":"2024-03-31T15:55:21.121071+0000","last_clean":"2024-03-31T15:55:21.121071+0000","last_became_active":"2024-03-31T15:55:17.562945+0000","last_became_peered":"2024-03-31T15:55:17.562945+0000","last_unstale":"2024-03-31T15:55:21.121071+0000","last_undegraded":"2024-03-31T15:55:21.121071+0000","last_fullsized":"2024-03-31T15:55:21.121071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120928+0000","last_change":"2024-03-31T15:55:18.557599+0000","last_active":"2024-03-31T15:55:21.120928+0000","last_peered":"2024-03-31T15:55:21.120928+0000","last_clean":"2024-03-31T15:55:21.120928+0000","last_became_active":"2024-03-31T15:55:18.557275+0000","last_became_peered":"2024-03-31T15:55:18.557275+0000","last_unstale":"2024-03-31T15:55:21.120928+0000","last_undegraded":"2024-03-31T15:55:21.120928+0000","last_fullsized":"2024-03-31T15:55:21.120928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550772+0000","last_change":"2024-03-31T15:55:17.562464+0000","last_active":"2024-03-31T15:55:19.550772+0000","last_peered":"2024-03-31T15:55:19.550772+0000","last_clean":"2024-03-31T15:55:19.550772+0000","last_became_active":"2024-03-31T15:55:17.562192+0000","last_became_peered":"2024-03-31T15:55:17.562192+0000","last_unstale":"2024-03-31T15:55:19.550772+0000","last_undegraded":"2024-03-31T15:55:19.550772+0000","last_fullsized":"2024-03-31T15:55:19.550772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:13.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550931+0000","last_change":"2024-03-31T15:55:18.572667+0000","last_active":"2024-03-31T15:55:19.550931+0000","last_peered":"2024-03-31T15:55:19.550931+0000","last_clean":"2024-03-31T15:55:19.550931+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.550931+0000","last_undegraded":"2024-03-31T15:55:19.550931+0000","last_fullsized":"2024-03-31T15:55:19.550931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:54:13.587576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930339+0000","last_change":"2024-03-31T15:55:17.564482+0000","last_active":"2024-03-31T15:55:20.930339+0000","last_peered":"2024-03-31T15:55:20.930339+0000","last_clean":"2024-03-31T15:55:20.930339+0000","last_became_active":"2024-03-31T15:55:17.564263+0000","last_became_peered":"2024-03-31T15:55:17.564263+0000","last_unstale":"2024-03-31T15:55:20.930339+0000","last_undegraded":"2024-03-31T15:55:20.930339+0000","last_fullsized":"2024-03-31T15:55:20.930339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583198+0000","last_change":"2024-03-31T15:55:18.573739+0000","last_active":"2024-03-31T15:55:18.583198+0000","last_peered":"2024-03-31T15:55:18.583198+0000","last_clean":"2024-03-31T15:55:18.583198+0000","last_became_active":"2024-03-31T15:55:18.573288+0000","last_became_peered":"2024-03-31T15:55:18.573288+0000","last_unstale":"2024-03-31T15:55:18.583198+0000","last_undegraded":"2024-03-31T15:55:18.583198+0000","last_fullsized":"2024-03-31T15:55:18.583198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:27.032794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550590+0000","last_change":"2024-03-31T15:55:17.561760+0000","last_active":"2024-03-31T15:55:19.550590+0000","last_peered":"2024-03-31T15:55:19.550590+0000","last_clean":"2024-03-31T15:55:19.550590+0000","last_became_active":"2024-03-31T15:55:17.561471+0000","last_became_peered":"2024-03-31T15:55:17.561471+0000","last_unstale":"2024-03-31T15:55:19.550590+0000","last_undegraded":"2024-03-31T15:55:19.550590+0000","last_fullsized":"2024-03-31T15:55:19.550590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:31:27.557663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837752+0000","last_change":"2024-03-31T15:55:18.557363+0000","last_active":"2024-03-31T15:55:20.837752+0000","last_peered":"2024-03-31T15:55:20.837752+0000","last_clean":"2024-03-31T15:55:20.837752+0000","last_became_active":"2024-03-31T15:55:18.557133+0000","last_became_peered":"2024-03-31T15:55:18.557133+0000","last_unstale":"2024-03-31T15:55:20.837752+0000","last_undegraded":"2024-03-31T15:55:20.837752+0000","last_fullsized":"2024-03-31T15:55:20.837752+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550455+0000","last_change":"2024-03-31T15:55:17.567522+0000","last_active":"2024-03-31T15:55:19.550455+0000","last_peered":"2024-03-31T15:55:19.550455+0000","last_clean":"2024-03-31T15:55:19.550455+0000","last_became_active":"2024-03-31T15:55:17.567296+0000","last_became_peered":"2024-03-31T15:55:17.567296+0000","last_unstale":"2024-03-31T15:55:19.550455+0000","last_undegraded":"2024-03-31T15:55:19.550455+0000","last_fullsized":"2024-03-31T15:55:19.550455+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:24:18.305487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553568+0000","last_change":"2024-03-31T15:55:18.578437+0000","last_active":"2024-03-31T15:55:19.553568+0000","last_peered":"2024-03-31T15:55:19.553568+0000","last_clean":"2024-03-31T15:55:19.553568+0000","last_became_active":"2024-03-31T15:55:18.578122+0000","last_became_peered":"2024-03-31T15:55:18.578122+0000","last_unstale":"2024-03-31T15:55:19.553568+0000","last_undegraded":"2024-03-31T15:55:19.553568+0000","last_fullsized":"2024-03-31T15:55:19.553568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:24.502072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554616+0000","last_change":"2024-03-31T15:55:17.569573+0000","last_active":"2024-03-31T15:55:18.554616+0000","last_peered":"2024-03-31T15:55:18.554616+0000","last_clean":"2024-03-31T15:55:18.554616+0000","last_became_active":"2024-03-31T15:55:17.569295+0000","last_became_peered":"2024-03-31T15:55:17.569295+0000","last_unstale":"2024-03-31T15:55:18.554616+0000","last_undegraded":"2024-03-31T15:55:18.554616+0000","last_fullsized":"2024-03-31T15:55:18.554616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:11.923571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548574+0000","last_change":"2024-03-31T15:55:18.581151+0000","last_active":"2024-03-31T15:55:19.548574+0000","last_peered":"2024-03-31T15:55:19.548574+0000","last_clean":"2024-03-31T15:55:19.548574+0000","last_became_active":"2024-03-31T15:55:18.580843+0000","last_became_peered":"2024-03-31T15:55:18.580843+0000","last_unstale":"2024-03-31T15:55:19.548574+0000","last_undegraded":"2024-03-31T15:55:19.548574+0000","last_fullsized":"2024-03-31T15:55:19.548574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550481+0000","last_change":"2024-03-31T15:55:17.561569+0000","last_active":"2024-03-31T15:55:19.550481+0000","last_peered":"2024-03-31T15:55:19.550481+0000","last_clean":"2024-03-31T15:55:19.550481+0000","last_became_active":"2024-03-31T15:55:17.560629+0000","last_became_peered":"2024-03-31T15:55:17.560629+0000","last_unstale":"2024-03-31T15:55:19.550481+0000","last_undegraded":"2024-03-31T15:55:19.550481+0000","last_fullsized":"2024-03-31T15:55:19.550481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:36.826694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930424+0000","last_change":"2024-03-31T15:55:18.569951+0000","last_active":"2024-03-31T15:55:20.930424+0000","last_peered":"2024-03-31T15:55:20.930424+0000","last_clean":"2024-03-31T15:55:20.930424+0000","last_became_active":"2024-03-31T15:55:18.569622+0000","last_became_peered":"2024-03-31T15:55:18.569622+0000","last_unstale":"2024-03-31T15:55:20.930424+0000","last_undegraded":"2024-03-31T15:55:20.930424+0000","last_fullsized":"2024-03-31T15:55:20.930424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554129+0000","last_change":"2024-03-31T15:55:17.558967+0000","last_active":"2024-03-31T15:55:19.554129+0000","last_peered":"2024-03-31T15:55:19.554129+0000","last_clean":"2024-03-31T15:55:19.554129+0000","last_became_active":"2024-03-31T15:55:17.558642+0000","last_became_peered":"2024-03-31T15:55:17.558642+0000","last_unstale":"2024-03-31T15:55:19.554129+0000","last_undegraded":"2024-03-31T15:55:19.554129+0000","last_fullsized":"2024-03-31T15:55:19.554129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:47.530373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583276+0000","last_change":"2024-03-31T15:55:18.576406+0000","last_active":"2024-03-31T15:55:18.583276+0000","last_peered":"2024-03-31T15:55:18.583276+0000","last_clean":"2024-03-31T15:55:18.583276+0000","last_became_active":"2024-03-31T15:55:18.575892+0000","last_became_peered":"2024-03-31T15:55:18.575892+0000","last_unstale":"2024-03-31T15:55:18.583276+0000","last_undegraded":"2024-03-31T15:55:18.583276+0000","last_fullsized":"2024-03-31T15:55:18.583276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:01.439448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.552589+0000","last_change":"2024-03-31T15:55:17.565054+0000","last_active":"2024-03-31T15:55:19.552589+0000","last_peered":"2024-03-31T15:55:19.552589+0000","last_clean":"2024-03-31T15:55:19.552589+0000","last_became_active":"2024-03-31T15:55:17.564845+0000","last_became_peered":"2024-03-31T15:55:17.564845+0000","last_unstale":"2024-03-31T15:55:19.552589+0000","last_undegraded":"2024-03-31T15:55:19.552589+0000","last_fullsized":"2024-03-31T15:55:19.552589+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:34:03.576042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548610+0000","last_change":"2024-03-31T15:55:18.572781+0000","last_active":"2024-03-31T15:55:19.548610+0000","last_peered":"2024-03-31T15:55:19.548610+0000","last_clean":"2024-03-31T15:55:19.548610+0000","last_became_active":"2024-03-31T15:55:18.572475+0000","last_became_peered":"2024-03-31T15:55:18.572475+0000","last_unstale":"2024-03-31T15:55:19.548610+0000","last_undegraded":"2024-03-31T15:55:19.548610+0000","last_fullsized":"2024-03-31T15:55:19.548610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:21.840147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838034+0000","last_change":"2024-03-31T15:55:17.552434+0000","last_active":"2024-03-31T15:55:20.838034+0000","last_peered":"2024-03-31T15:55:20.838034+0000","last_clean":"2024-03-31T15:55:20.838034+0000","last_became_active":"2024-03-31T15:55:17.552137+0000","last_became_peered":"2024-03-31T15:55:17.552137+0000","last_unstale":"2024-03-31T15:55:20.838034+0000","last_undegraded":"2024-03-31T15:55:20.838034+0000","last_fullsized":"2024-03-31T15:55:20.838034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837915+0000","last_change":"2024-03-31T15:55:18.567217+0000","last_active":"2024-03-31T15:55:20.837915+0000","last_peered":"2024-03-31T15:55:20.837915+0000","last_clean":"2024-03-31T15:55:20.837915+0000","last_became_active":"2024-03-31T15:55:18.566527+0000","last_became_peered":"2024-03-31T15:55:18.566527+0000","last_unstale":"2024-03-31T15:55:20.837915+0000","last_undegraded":"2024-03-31T15:55:20.837915+0000","last_fullsized":"2024-03-31T15:55:20.837915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837837+0000","last_change":"2024-03-31T15:55:17.569970+0000","last_active":"2024-03-31T15:55:20.837837+0000","last_peered":"2024-03-31T15:55:20.837837+0000","last_clean":"2024-03-31T15:55:20.837837+0000","last_became_active":"2024-03-31T15:55:17.569178+0000","last_became_peered":"2024-03-31T15:55:17.569178+0000","last_unstale":"2024-03-31T15:55:20.837837+0000","last_undegraded":"2024-03-31T15:55:20.837837+0000","last_fullsized":"2024-03-31T15:55:20.837837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:18.576213+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:18.575930+0000","last_became_peered":"2024-03-31T15:55:18.575930+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547246+0000","last_change":"2024-03-31T15:55:17.569351+0000","last_active":"2024-03-31T15:55:19.547246+0000","last_peered":"2024-03-31T15:55:19.547246+0000","last_clean":"2024-03-31T15:55:19.547246+0000","last_became_active":"2024-03-31T15:55:17.569079+0000","last_became_peered":"2024-03-31T15:55:17.569079+0000","last_unstale":"2024-03-31T15:55:19.547246+0000","last_undegraded":"2024-03-31T15:55:19.547246+0000","last_fullsized":"2024-03-31T15:55:19.547246+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120705+0000","last_change":"2024-03-31T15:55:18.572994+0000","last_active":"2024-03-31T15:55:21.120705+0000","last_peered":"2024-03-31T15:55:21.120705+0000","last_clean":"2024-03-31T15:55:21.120705+0000","last_became_active":"2024-03-31T15:55:18.572692+0000","last_became_peered":"2024-03-31T15:55:18.572692+0000","last_unstale":"2024-03-31T15:55:21.120705+0000","last_undegraded":"2024-03-31T15:55:21.120705+0000","last_fullsized":"2024-03-31T15:55:21.120705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120545+0000","last_change":"2024-03-31T15:55:17.563312+0000","last_active":"2024-03-31T15:55:21.120545+0000","last_peered":"2024-03-31T15:55:21.120545+0000","last_clean":"2024-03-31T15:55:21.120545+0000","last_became_active":"2024-03-31T15:55:17.562982+0000","last_became_peered":"2024-03-31T15:55:17.562982+0000","last_unstale":"2024-03-31T15:55:21.120545+0000","last_undegraded":"2024-03-31T15:55:21.120545+0000","last_fullsized":"2024-03-31T15:55:21.120545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583180+0000","last_change":"2024-03-31T15:55:18.575859+0000","last_active":"2024-03-31T15:55:18.583180+0000","last_peered":"2024-03-31T15:55:18.583180+0000","last_clean":"2024-03-31T15:55:18.583180+0000","last_became_active":"2024-03-31T15:55:18.575507+0000","last_became_peered":"2024-03-31T15:55:18.575507+0000","last_unstale":"2024-03-31T15:55:18.583180+0000","last_undegraded":"2024-03-31T15:55:18.583180+0000","last_fullsized":"2024-03-31T15:55:18.583180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:31:28.813601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547913+0000","last_change":"2024-03-31T15:55:17.559862+0000","last_active":"2024-03-31T15:55:19.547913+0000","last_peered":"2024-03-31T15:55:19.547913+0000","last_clean":"2024-03-31T15:55:19.547913+0000","last_became_active":"2024-03-31T15:55:17.558616+0000","last_became_peered":"2024-03-31T15:55:17.558616+0000","last_unstale":"2024-03-31T15:55:19.547913+0000","last_undegraded":"2024-03-31T15:55:19.547913+0000","last_fullsized":"2024-03-31T15:55:19.547913+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:28.724128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553996+0000","last_change":"2024-03-31T15:55:18.582994+0000","last_active":"2024-03-31T15:55:19.553996+0000","last_peered":"2024-03-31T15:55:19.553996+0000","last_clean":"2024-03-31T15:55:19.553996+0000","last_became_active":"2024-03-31T15:55:18.580575+0000","last_became_peered":"2024-03-31T15:55:18.580575+0000","last_unstale":"2024-03-31T15:55:19.553996+0000","last_undegraded":"2024-03-31T15:55:19.553996+0000","last_fullsized":"2024-03-31T15:55:19.553996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:59.330574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554197+0000","last_change":"2024-03-31T15:55:17.570487+0000","last_active":"2024-03-31T15:55:18.554197+0000","last_peered":"2024-03-31T15:55:18.554197+0000","last_clean":"2024-03-31T15:55:18.554197+0000","last_became_active":"2024-03-31T15:55:17.570204+0000","last_became_peered":"2024-03-31T15:55:17.570204+0000","last_unstale":"2024-03-31T15:55:18.554197+0000","last_undegraded":"2024-03-31T15:55:18.554197+0000","last_fullsized":"2024-03-31T15:55:18.554197+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:34.516949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.553800+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:55:18.553800+0000","last_peered":"2024-03-31T15:55:18.553800+0000","last_clean":"2024-03-31T15:55:18.553800+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:55:18.553800+0000","last_undegraded":"2024-03-31T15:55:18.553800+0000","last_fullsized":"2024-03-31T15:55:18.553800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":192513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509464,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":450560,"data_stored":192043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509465,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":185329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":782333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":186885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":185309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":777289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:55:23.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-31T15:55:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 Processor -- start 2024-03-31T15:55:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 -- start start 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c06b820 0x7f9e3c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3c0b34f0 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3c0b3ab0 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e42400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c141ba0 con 0x7f9e3c06b820 2024-03-31T15:55:23.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50872/0 (socket says 172.21.15.90:50872) 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 -- 172.21.15.90:0/361735781 learned_addr learned my addr 172.21.15.90:0/361735781 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3be00640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c06b820 0x7f9e3c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 -- 172.21.15.90:0/361735781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c06b820 msgr2=0x7f9e3c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c06b820 0x7f9e3c06bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 -- 172.21.15.90:0/361735781 >> v1:172.21.15.102:6789/0 conn(0x7f9e3c0cc9d0 legacy=0x7f9e3c142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.644+0000 7f9e3aa00640 1 -- 172.21.15.90:0/361735781 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c0b37d0 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e3aa00640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3b3d264d6b23c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e3a000640 1 -- 172.21.15.90:0/361735781 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280b1070 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e3a000640 1 -- 172.21.15.90:0/361735781 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e280a8050 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e3a000640 1 -- 172.21.15.90:0/361735781 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280ac860 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- 172.21.15.90:0/361735781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0ccdb0 msgr2=0x7f9e3c0b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009870 tx=0x7f9e2801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- 172.21.15.90:0/361735781 shutdown_connections 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c06b820 0x7f9e3c06bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 --2- 172.21.15.90:0/361735781 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0ccdb0 0x7f9e3c0b2b80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- 172.21.15.90:0/361735781 >> 172.21.15.90:0/361735781 conn(0x7f9e3c0b1c80 msgr2=0x7f9e3c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- 172.21.15.90:0/361735781 shutdown_connections 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- 172.21.15.90:0/361735781 wait complete. 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 Processor -- start 2024-03-31T15:55:23.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.648+0000 7f9e42400640 1 -- start start 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e3c06b820 0x7f9e3c1fb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c0ccdb0 0x7f9e3c1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cf0c0 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0cef40 con 0x7f9e3c06b820 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c15c620 con 0x7f9e3c0ccdb0 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50874/0 (socket says 172.21.15.90:50874) 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 -- 172.21.15.90:0/3331883328 learned_addr learned my addr 172.21.15.90:0/3331883328 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:23.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3b400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e3c06b820 0x7f9e3c1fb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3be00640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c0ccdb0 0x7f9e3c1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 -- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c0ccdb0 msgr2=0x7f9e3c1f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c0ccdb0 0x7f9e3c1f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 -- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e3c06b820 msgr2=0x7f9e3c1fb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e3c06b820 0x7f9e3c1fb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 -- 172.21.15.90:0/3331883328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c1fb870 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e3aa00640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9e2801ab80 tx=0x7f9e28019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280bd070 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e280b7630 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e280b1040 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3c1f7e80 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3c1fab50 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.652+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3c1fa940 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.656+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9e280c0070 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.656+0000 7f9e33400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 0x7f9e1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:23.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.656+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9e281206e0 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.656+0000 7f9e3b400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 0x7f9e1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:23.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.656+0000 7f9e3b400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 0x7f9e1c06a840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c001810 tx=0x7f9e2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:23.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.660+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9e28016150 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.872+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f9e3c0686d0 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e33400640 1 -- 172.21.15.90:0/3331883328 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1013 (secure 0 0 0) 0x7f9e280f04c0 con 0x7f9e3c0cc9d0 2024-03-31T15:55:23.868 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:23.869 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1711900415,"num_in_osds":8,"osd_in_since":1711900406,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":226115584,"bytes_avail":772867997696,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5502}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-31T15:54:55.233300+0000","services":{}},"progress_events":{}} 2024-03-31T15:55:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 msgr2=0x7f9e1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 0x7f9e1c06a840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c001810 tx=0x7f9e2c009040 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 msgr2=0x7f9e3c152a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9e2801ab80 tx=0x7f9e28019270 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 shutdown_connections 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9e1c068380 0x7f9e1c06a840 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9e3c0ccdb0 0x7f9e3c1f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9e3c0cc9d0 0x7f9e3c152a60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 --2- 172.21.15.90:0/3331883328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e3c06b820 0x7f9e3c1fb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 >> 172.21.15.90:0/3331883328 conn(0x7f9e3c0b1c80 msgr2=0x7f9e3c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 shutdown_connections 2024-03-31T15:55:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:23.876+0000 7f9e42400640 1 -- 172.21.15.90:0/3331883328 wait complete. 2024-03-31T15:55:23.897 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-31T15:55:23.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:55:24.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 Processor -- start 2024-03-31T15:55:24.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 -- start start 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a0815e1b0 0x7f9a08162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9a0806c7b0 0x7f9a0806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0815f780 con 0x7f9a0806d5f0 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9a081622b0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a0de00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9a0806da20 con 0x7f9a0806c7b0 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a0815e1b0 0x7f9a08162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a0815e1b0 0x7f9a08162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50888/0 (socket says 172.21.15.90:50888) 2024-03-31T15:55:24.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 -- 172.21.15.90:0/3375274715 learned_addr learned my addr 172.21.15.90:0/3375274715 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f99ffe00640 1 --2- 172.21.15.90:0/3375274715 >> v2:172.21.15.102:3300/0 conn(0x7f9a0806c7b0 0x7f9a0806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 -- 172.21.15.90:0/3375274715 >> v2:172.21.15.102:3300/0 conn(0x7f9a0806c7b0 msgr2=0x7f9a0806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 --2- 172.21.15.90:0/3375274715 >> v2:172.21.15.102:3300/0 conn(0x7f9a0806c7b0 0x7f9a0806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 -- 172.21.15.90:0/3375274715 >> v1:172.21.15.102:6789/0 conn(0x7f9a0806d5f0 legacy=0x7f9a0815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:24.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.040+0000 7f9a06a00640 1 -- 172.21.15.90:0/3375274715 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0815f1c0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a06a00640 1 --2- 172.21.15.90:0/3375274715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a08162740 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9a0806ed50 tx=0x7f99f801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6251b1a839c235e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a06000640 1 -- 172.21.15.90:0/3375274715 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80ab620 con 0x7f9a0815e1b0 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a06000640 1 -- 172.21.15.90:0/3375274715 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f80b0070 con 0x7f9a0815e1b0 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a06000640 1 -- 172.21.15.90:0/3375274715 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80ab9e0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 -- 172.21.15.90:0/3375274715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 msgr2=0x7f9a08162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 --2- 172.21.15.90:0/3375274715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a08162740 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9a0806ed50 tx=0x7f99f801c420 comp rx=0 tx=0).stop 2024-03-31T15:55:24.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 -- 172.21.15.90:0/3375274715 shutdown_connections 2024-03-31T15:55:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 --2- 172.21.15.90:0/3375274715 >> v2:172.21.15.102:3300/0 conn(0x7f9a0806c7b0 0x7f9a0806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 --2- 172.21.15.90:0/3375274715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a08162740 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 -- 172.21.15.90:0/3375274715 >> 172.21.15.90:0/3375274715 conn(0x7f9a0809aa40 msgr2=0x7f9a0809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 -- 172.21.15.90:0/3375274715 shutdown_connections 2024-03-31T15:55:24.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.044+0000 7f9a0de00640 1 -- 172.21.15.90:0/3375274715 wait complete. 2024-03-31T15:55:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 Processor -- start 2024-03-31T15:55:24.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- start start 2024-03-31T15:55:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a0806c7b0 0x7f9a080c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a0806d5f0 0x7f9a080c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d2940 con 0x7f9a0815e1b0 2024-03-31T15:55:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d27c0 con 0x7f9a0806d5f0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9a080d2ac0 con 0x7f9a0806c7b0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50890/0 (socket says 172.21.15.90:50890) 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 -- 172.21.15.90:0/2175644271 learned_addr learned my addr 172.21.15.90:0/2175644271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f99ffe00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a0806d5f0 0x7f9a080c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a06a00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a0806c7b0 0x7f9a080c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 -- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a0806c7b0 msgr2=0x7f9a080c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a0806c7b0 0x7f9a080c5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 -- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a0806d5f0 msgr2=0x7f9a080c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a0806d5f0 0x7f9a080c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 -- 172.21.15.90:0/2175644271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a080c62d0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a07400640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b970 tx=0x7f9a0000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:24.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00019070 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0000cb40 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00014440 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a080c65b0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.048+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a080c6b70 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a080686d0 con 0x7f9a0815e1b0 2024-03-31T15:55:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a0001c070 con 0x7f9a0815e1b0 2024-03-31T15:55:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f99ff400640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 0x7f99e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9a00086810 con 0x7f9a0815e1b0 2024-03-31T15:55:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f9a06a00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 0x7f99e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:24.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.052+0000 7f9a06a00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 0x7f99e406a840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f99f80af000 tx=0x7f99f80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:24.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.056+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a00056550 con 0x7f9a0815e1b0 2024-03-31T15:55:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.212+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a080b7b70 con 0x7f99e4068380 2024-03-31T15:55:24.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.216+0000 7f99ff400640 1 -- 172.21.15.90:0/2175644271 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7f9a080b7b70 con 0x7f99e4068380 2024-03-31T15:55:24.214 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 msgr2=0x7f99e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 0x7f99e406a840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f99f80af000 tx=0x7f99f80a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 msgr2=0x7f9a0809d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b970 tx=0x7f9a0000be30 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 shutdown_connections 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f99e4068380 0x7f99e406a840 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a0815e1b0 0x7f9a0809d2c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a0806d5f0 0x7f9a080c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 --2- 172.21.15.90:0/2175644271 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a0806c7b0 0x7f9a080c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 >> 172.21.15.90:0/2175644271 conn(0x7f9a0809aa40 msgr2=0x7f9a0815fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:24.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 shutdown_connections 2024-03-31T15:55:24.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:24.220+0000 7f9a0de00640 1 -- 172.21.15.90:0/2175644271 wait complete. 2024-03-31T15:55:24.238 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-03-31T15:55:23.239082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220816,"kb_used_data":4632,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753904,"statfs":{"total":773094113280,"available":772867997696,"internally_reserved":0,"allocated":4743168,"data_stored":2694236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001120"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548723+0000","last_change":"2024-03-31T15:55:18.569836+0000","last_active":"2024-03-31T15:55:19.548723+0000","last_peered":"2024-03-31T15:55:19.548723+0000","last_clean":"2024-03-31T15:55:19.548723+0000","last_became_active":"2024-03-31T15:55:18.569542+0000","last_became_peered":"2024-03-31T15:55:18.569542+0000","last_unstale":"2024-03-31T15:55:19.548723+0000","last_undegraded":"2024-03-31T15:55:19.548723+0000","last_fullsized":"2024-03-31T15:55:19.548723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:52.818181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548259+0000","last_change":"2024-03-31T15:55:18.570636+0000","last_active":"2024-03-31T15:55:19.548259+0000","last_peered":"2024-03-31T15:55:19.548259+0000","last_clean":"2024-03-31T15:55:19.548259+0000","last_became_active":"2024-03-31T15:55:18.570347+0000","last_became_peered":"2024-03-31T15:55:18.570347+0000","last_unstale":"2024-03-31T15:55:19.548259+0000","last_undegraded":"2024-03-31T15:55:19.548259+0000","last_fullsized":"2024-03-31T15:55:19.548259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838065+0000","last_change":"2024-03-31T15:55:18.576959+0000","last_active":"2024-03-31T15:55:20.838065+0000","last_peered":"2024-03-31T15:55:20.838065+0000","last_clean":"2024-03-31T15:55:20.838065+0000","last_became_active":"2024-03-31T15:55:18.576703+0000","last_became_peered":"2024-03-31T15:55:18.576703+0000","last_unstale":"2024-03-31T15:55:20.838065+0000","last_undegraded":"2024-03-31T15:55:20.838065+0000","last_fullsized":"2024-03-31T15:55:20.838065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550774+0000","last_change":"2024-03-31T15:55:18.558889+0000","last_active":"2024-03-31T15:55:19.550774+0000","last_peered":"2024-03-31T15:55:19.550774+0000","last_clean":"2024-03-31T15:55:19.550774+0000","last_became_active":"2024-03-31T15:55:18.558620+0000","last_became_peered":"2024-03-31T15:55:18.558620+0000","last_unstale":"2024-03-31T15:55:19.550774+0000","last_undegraded":"2024-03-31T15:55:19.550774+0000","last_fullsized":"2024-03-31T15:55:19.550774+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:16.723939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550456+0000","last_change":"2024-03-31T15:55:18.552183+0000","last_active":"2024-03-31T15:55:19.550456+0000","last_peered":"2024-03-31T15:55:19.550456+0000","last_clean":"2024-03-31T15:55:19.550456+0000","last_became_active":"2024-03-31T15:55:18.551886+0000","last_became_peered":"2024-03-31T15:55:18.551886+0000","last_unstale":"2024-03-31T15:55:19.550456+0000","last_undegraded":"2024-03-31T15:55:19.550456+0000","last_fullsized":"2024-03-31T15:55:19.550456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:53:07.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930104+0000","last_change":"2024-03-31T15:55:18.577900+0000","last_active":"2024-03-31T15:55:20.930104+0000","last_peered":"2024-03-31T15:55:20.930104+0000","last_clean":"2024-03-31T15:55:20.930104+0000","last_became_active":"2024-03-31T15:55:18.577638+0000","last_became_peered":"2024-03-31T15:55:18.577638+0000","last_unstale":"2024-03-31T15:55:20.930104+0000","last_undegraded":"2024-03-31T15:55:20.930104+0000","last_fullsized":"2024-03-31T15:55:20.930104+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553456+0000","last_change":"2024-03-31T15:55:18.569983+0000","last_active":"2024-03-31T15:55:19.553456+0000","last_peered":"2024-03-31T15:55:19.553456+0000","last_clean":"2024-03-31T15:55:19.553456+0000","last_became_active":"2024-03-31T15:55:18.569706+0000","last_became_peered":"2024-03-31T15:55:18.569706+0000","last_unstale":"2024-03-31T15:55:19.553456+0000","last_undegraded":"2024-03-31T15:55:19.553456+0000","last_fullsized":"2024-03-31T15:55:19.553456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:17:54.427505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547884+0000","last_change":"2024-03-31T15:55:18.572173+0000","last_active":"2024-03-31T15:55:19.547884+0000","last_peered":"2024-03-31T15:55:19.547884+0000","last_clean":"2024-03-31T15:55:19.547884+0000","last_became_active":"2024-03-31T15:55:18.571909+0000","last_became_peered":"2024-03-31T15:55:18.571909+0000","last_unstale":"2024-03-31T15:55:19.547884+0000","last_undegraded":"2024-03-31T15:55:19.547884+0000","last_fullsized":"2024-03-31T15:55:19.547884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838187+0000","last_change":"2024-03-31T15:55:18.577912+0000","last_active":"2024-03-31T15:55:20.838187+0000","last_peered":"2024-03-31T15:55:20.838187+0000","last_clean":"2024-03-31T15:55:20.838187+0000","last_became_active":"2024-03-31T15:55:18.574214+0000","last_became_peered":"2024-03-31T15:55:18.574214+0000","last_unstale":"2024-03-31T15:55:20.838187+0000","last_undegraded":"2024-03-31T15:55:20.838187+0000","last_fullsized":"2024-03-31T15:55:20.838187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:48.143140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553957+0000","last_change":"2024-03-31T15:55:18.567327+0000","last_active":"2024-03-31T15:55:19.553957+0000","last_peered":"2024-03-31T15:55:19.553957+0000","last_clean":"2024-03-31T15:55:19.553957+0000","last_became_active":"2024-03-31T15:55:18.567046+0000","last_became_peered":"2024-03-31T15:55:18.567046+0000","last_unstale":"2024-03-31T15:55:19.553957+0000","last_undegraded":"2024-03-31T15:55:19.553957+0000","last_fullsized":"2024-03-31T15:55:19.553957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:21.180135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553799+0000","last_change":"2024-03-31T15:55:18.562702+0000","last_active":"2024-03-31T15:55:19.553799+0000","last_peered":"2024-03-31T15:55:19.553799+0000","last_clean":"2024-03-31T15:55:19.553799+0000","last_became_active":"2024-03-31T15:55:18.562447+0000","last_became_peered":"2024-03-31T15:55:18.562447+0000","last_unstale":"2024-03-31T15:55:19.553799+0000","last_undegraded":"2024-03-31T15:55:19.553799+0000","last_fullsized":"2024-03-31T15:55:19.553799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:59:15.558471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838024+0000","last_change":"2024-03-31T15:55:18.562485+0000","last_active":"2024-03-31T15:55:20.838024+0000","last_peered":"2024-03-31T15:55:20.838024+0000","last_clean":"2024-03-31T15:55:20.838024+0000","last_became_active":"2024-03-31T15:55:18.562260+0000","last_became_peered":"2024-03-31T15:55:18.562260+0000","last_unstale":"2024-03-31T15:55:20.838024+0000","last_undegraded":"2024-03-31T15:55:20.838024+0000","last_fullsized":"2024-03-31T15:55:20.838024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553436+0000","last_change":"2024-03-31T15:55:18.564320+0000","last_active":"2024-03-31T15:55:19.553436+0000","last_peered":"2024-03-31T15:55:19.553436+0000","last_clean":"2024-03-31T15:55:19.553436+0000","last_became_active":"2024-03-31T15:55:18.564064+0000","last_became_peered":"2024-03-31T15:55:18.564064+0000","last_unstale":"2024-03-31T15:55:19.553436+0000","last_undegraded":"2024-03-31T15:55:19.553436+0000","last_fullsized":"2024-03-31T15:55:19.553436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:06.438509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120606+0000","last_change":"2024-03-31T15:55:18.581725+0000","last_active":"2024-03-31T15:55:21.120606+0000","last_peered":"2024-03-31T15:55:21.120606+0000","last_clean":"2024-03-31T15:55:21.120606+0000","last_became_active":"2024-03-31T15:55:18.581115+0000","last_became_peered":"2024-03-31T15:55:18.581115+0000","last_unstale":"2024-03-31T15:55:21.120606+0000","last_undegraded":"2024-03-31T15:55:21.120606+0000","last_fullsized":"2024-03-31T15:55:21.120606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553656+0000","last_change":"2024-03-31T15:55:18.562134+0000","last_active":"2024-03-31T15:55:19.553656+0000","last_peered":"2024-03-31T15:55:19.553656+0000","last_clean":"2024-03-31T15:55:19.553656+0000","last_became_active":"2024-03-31T15:55:18.559688+0000","last_became_peered":"2024-03-31T15:55:18.559688+0000","last_unstale":"2024-03-31T15:55:19.553656+0000","last_undegraded":"2024-03-31T15:55:19.553656+0000","last_fullsized":"2024-03-31T15:55:19.553656+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:57.153194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580155+0000","last_change":"2024-03-31T15:55:18.573440+0000","last_active":"2024-03-31T15:55:18.580155+0000","last_peered":"2024-03-31T15:55:18.580155+0000","last_clean":"2024-03-31T15:55:18.580155+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:18.580155+0000","last_undegraded":"2024-03-31T15:55:18.580155+0000","last_fullsized":"2024-03-31T15:55:18.580155+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:44.338598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.579763+0000","last_change":"2024-03-31T15:55:18.571940+0000","last_active":"2024-03-31T15:55:18.579763+0000","last_peered":"2024-03-31T15:55:18.579763+0000","last_clean":"2024-03-31T15:55:18.579763+0000","last_became_active":"2024-03-31T15:55:18.571687+0000","last_became_peered":"2024-03-31T15:55:18.571687+0000","last_unstale":"2024-03-31T15:55:18.579763+0000","last_undegraded":"2024-03-31T15:55:18.579763+0000","last_fullsized":"2024-03-31T15:55:18.579763+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:24.106111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547503+0000","last_change":"2024-03-31T15:55:18.581100+0000","last_active":"2024-03-31T15:55:19.547503+0000","last_peered":"2024-03-31T15:55:19.547503+0000","last_clean":"2024-03-31T15:55:19.547503+0000","last_became_active":"2024-03-31T15:55:18.580799+0000","last_became_peered":"2024-03-31T15:55:18.580799+0000","last_unstale":"2024-03-31T15:55:19.547503+0000","last_undegraded":"2024-03-31T15:55:19.547503+0000","last_fullsized":"2024-03-31T15:55:19.547503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550838+0000","last_change":"2024-03-31T15:55:18.565501+0000","last_active":"2024-03-31T15:55:19.550838+0000","last_peered":"2024-03-31T15:55:19.550838+0000","last_clean":"2024-03-31T15:55:19.550838+0000","last_became_active":"2024-03-31T15:55:18.565247+0000","last_became_peered":"2024-03-31T15:55:18.565247+0000","last_unstale":"2024-03-31T15:55:19.550838+0000","last_undegraded":"2024-03-31T15:55:19.550838+0000","last_fullsized":"2024-03-31T15:55:19.550838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:55.707922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:18.576219+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:18.575905+0000","last_became_peered":"2024-03-31T15:55:18.575905+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553636+0000","last_change":"2024-03-31T15:55:18.559315+0000","last_active":"2024-03-31T15:55:19.553636+0000","last_peered":"2024-03-31T15:55:19.553636+0000","last_clean":"2024-03-31T15:55:19.553636+0000","last_became_active":"2024-03-31T15:55:18.558870+0000","last_became_peered":"2024-03-31T15:55:18.558870+0000","last_unstale":"2024-03-31T15:55:19.553636+0000","last_undegraded":"2024-03-31T15:55:19.553636+0000","last_fullsized":"2024-03-31T15:55:19.553636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:45.294220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121007+0000","last_change":"2024-03-31T15:55:18.571748+0000","last_active":"2024-03-31T15:55:21.121007+0000","last_peered":"2024-03-31T15:55:21.121007+0000","last_clean":"2024-03-31T15:55:21.121007+0000","last_became_active":"2024-03-31T15:55:18.571445+0000","last_became_peered":"2024-03-31T15:55:18.571445+0000","last_unstale":"2024-03-31T15:55:21.121007+0000","last_undegraded":"2024-03-31T15:55:21.121007+0000","last_fullsized":"2024-03-31T15:55:21.121007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583121+0000","last_change":"2024-03-31T15:55:18.575290+0000","last_active":"2024-03-31T15:55:18.583121+0000","last_peered":"2024-03-31T15:55:18.583121+0000","last_clean":"2024-03-31T15:55:18.583121+0000","last_became_active":"2024-03-31T15:55:18.574928+0000","last_became_peered":"2024-03-31T15:55:18.574928+0000","last_unstale":"2024-03-31T15:55:18.583121+0000","last_undegraded":"2024-03-31T15:55:18.583121+0000","last_fullsized":"2024-03-31T15:55:18.583121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:55:04.640612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550628+0000","last_change":"2024-03-31T15:55:18.563439+0000","last_active":"2024-03-31T15:55:19.550628+0000","last_peered":"2024-03-31T15:55:19.550628+0000","last_clean":"2024-03-31T15:55:19.550628+0000","last_became_active":"2024-03-31T15:55:18.563160+0000","last_became_peered":"2024-03-31T15:55:18.563160+0000","last_unstale":"2024-03-31T15:55:19.550628+0000","last_undegraded":"2024-03-31T15:55:19.550628+0000","last_fullsized":"2024-03-31T15:55:19.550628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:33.025431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547597+0000","last_change":"2024-03-31T15:55:18.582358+0000","last_active":"2024-03-31T15:55:19.547597+0000","last_peered":"2024-03-31T15:55:19.547597+0000","last_clean":"2024-03-31T15:55:19.547597+0000","last_became_active":"2024-03-31T15:55:18.581841+0000","last_became_peered":"2024-03-31T15:55:18.581841+0000","last_unstale":"2024-03-31T15:55:19.547597+0000","last_undegraded":"2024-03-31T15:55:19.547597+0000","last_fullsized":"2024-03-31T15:55:19.547597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930482+0000","last_change":"2024-03-31T15:55:18.570686+0000","last_active":"2024-03-31T15:55:20.930482+0000","last_peered":"2024-03-31T15:55:20.930482+0000","last_clean":"2024-03-31T15:55:20.930482+0000","last_became_active":"2024-03-31T15:55:18.570424+0000","last_became_peered":"2024-03-31T15:55:18.570424+0000","last_unstale":"2024-03-31T15:55:20.930482+0000","last_undegraded":"2024-03-31T15:55:20.930482+0000","last_fullsized":"2024-03-31T15:55:20.930482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547692+0000","last_change":"2024-03-31T15:55:18.574827+0000","last_active":"2024-03-31T15:55:19.547692+0000","last_peered":"2024-03-31T15:55:19.547692+0000","last_clean":"2024-03-31T15:55:19.547692+0000","last_became_active":"2024-03-31T15:55:18.574539+0000","last_became_peered":"2024-03-31T15:55:18.574539+0000","last_unstale":"2024-03-31T15:55:19.547692+0000","last_undegraded":"2024-03-31T15:55:19.547692+0000","last_fullsized":"2024-03-31T15:55:19.547692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553659+0000","last_change":"2024-03-31T15:55:18.581864+0000","last_active":"2024-03-31T15:55:19.553659+0000","last_peered":"2024-03-31T15:55:19.553659+0000","last_clean":"2024-03-31T15:55:19.553659+0000","last_became_active":"2024-03-31T15:55:18.579446+0000","last_became_peered":"2024-03-31T15:55:18.579446+0000","last_unstale":"2024-03-31T15:55:19.553659+0000","last_undegraded":"2024-03-31T15:55:19.553659+0000","last_fullsized":"2024-03-31T15:55:19.553659+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:21.190346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547421+0000","last_change":"2024-03-31T15:55:18.581228+0000","last_active":"2024-03-31T15:55:19.547421+0000","last_peered":"2024-03-31T15:55:19.547421+0000","last_clean":"2024-03-31T15:55:19.547421+0000","last_became_active":"2024-03-31T15:55:18.580930+0000","last_became_peered":"2024-03-31T15:55:18.580930+0000","last_unstale":"2024-03-31T15:55:19.547421+0000","last_undegraded":"2024-03-31T15:55:19.547421+0000","last_fullsized":"2024-03-31T15:55:19.547421+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:48.538215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547145+0000","last_change":"2024-03-31T15:55:18.576569+0000","last_active":"2024-03-31T15:55:19.547145+0000","last_peered":"2024-03-31T15:55:19.547145+0000","last_clean":"2024-03-31T15:55:19.547145+0000","last_became_active":"2024-03-31T15:55:18.576267+0000","last_became_peered":"2024-03-31T15:55:18.576267+0000","last_unstale":"2024-03-31T15:55:19.547145+0000","last_undegraded":"2024-03-31T15:55:19.547145+0000","last_fullsized":"2024-03-31T15:55:19.547145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:02.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583094+0000","last_change":"2024-03-31T15:55:18.574506+0000","last_active":"2024-03-31T15:55:18.583094+0000","last_peered":"2024-03-31T15:55:18.583094+0000","last_clean":"2024-03-31T15:55:18.583094+0000","last_became_active":"2024-03-31T15:55:18.570090+0000","last_became_peered":"2024-03-31T15:55:18.570090+0000","last_unstale":"2024-03-31T15:55:18.583094+0000","last_undegraded":"2024-03-31T15:55:18.583094+0000","last_fullsized":"2024-03-31T15:55:18.583094+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:23:14.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930154+0000","last_change":"2024-03-31T15:55:18.560243+0000","last_active":"2024-03-31T15:55:20.930154+0000","last_peered":"2024-03-31T15:55:20.930154+0000","last_clean":"2024-03-31T15:55:20.930154+0000","last_became_active":"2024-03-31T15:55:18.559913+0000","last_became_peered":"2024-03-31T15:55:18.559913+0000","last_unstale":"2024-03-31T15:55:20.930154+0000","last_undegraded":"2024-03-31T15:55:20.930154+0000","last_fullsized":"2024-03-31T15:55:20.930154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580312+0000","last_change":"2024-03-31T15:55:18.570263+0000","last_active":"2024-03-31T15:55:18.580312+0000","last_peered":"2024-03-31T15:55:18.580312+0000","last_clean":"2024-03-31T15:55:18.580312+0000","last_became_active":"2024-03-31T15:55:18.569923+0000","last_became_peered":"2024-03-31T15:55:18.569923+0000","last_unstale":"2024-03-31T15:55:18.580312+0000","last_undegraded":"2024-03-31T15:55:18.580312+0000","last_fullsized":"2024-03-31T15:55:18.580312+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:29.731970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:17.569534+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:17.568983+0000","last_became_peered":"2024-03-31T15:55:17.568983+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930469+0000","last_change":"2024-03-31T15:55:18.575181+0000","last_active":"2024-03-31T15:55:20.930469+0000","last_peered":"2024-03-31T15:55:20.930469+0000","last_clean":"2024-03-31T15:55:20.930469+0000","last_became_active":"2024-03-31T15:55:18.574878+0000","last_became_peered":"2024-03-31T15:55:18.574878+0000","last_unstale":"2024-03-31T15:55:20.930469+0000","last_undegraded":"2024-03-31T15:55:20.930469+0000","last_fullsized":"2024-03-31T15:55:20.930469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838089+0000","last_change":"2024-03-31T15:55:17.561836+0000","last_active":"2024-03-31T15:55:20.838089+0000","last_peered":"2024-03-31T15:55:20.838089+0000","last_clean":"2024-03-31T15:55:20.838089+0000","last_became_active":"2024-03-31T15:55:17.561561+0000","last_became_peered":"2024-03-31T15:55:17.561561+0000","last_unstale":"2024-03-31T15:55:20.838089+0000","last_undegraded":"2024-03-31T15:55:20.838089+0000","last_fullsized":"2024-03-31T15:55:20.838089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121107+0000","last_change":"2024-03-31T15:55:18.580605+0000","last_active":"2024-03-31T15:55:21.121107+0000","last_peered":"2024-03-31T15:55:21.121107+0000","last_clean":"2024-03-31T15:55:21.121107+0000","last_became_active":"2024-03-31T15:55:18.580334+0000","last_became_peered":"2024-03-31T15:55:18.580334+0000","last_unstale":"2024-03-31T15:55:21.121107+0000","last_undegraded":"2024-03-31T15:55:21.121107+0000","last_fullsized":"2024-03-31T15:55:21.121107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930625+0000","last_change":"2024-03-31T15:55:17.566422+0000","last_active":"2024-03-31T15:55:20.930625+0000","last_peered":"2024-03-31T15:55:20.930625+0000","last_clean":"2024-03-31T15:55:20.930625+0000","last_became_active":"2024-03-31T15:55:17.566154+0000","last_became_peered":"2024-03-31T15:55:17.566154+0000","last_unstale":"2024-03-31T15:55:20.930625+0000","last_undegraded":"2024-03-31T15:55:20.930625+0000","last_fullsized":"2024-03-31T15:55:20.930625+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:42:15.816385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548666+0000","last_change":"2024-03-31T15:55:18.583489+0000","last_active":"2024-03-31T15:55:19.548666+0000","last_peered":"2024-03-31T15:55:19.548666+0000","last_clean":"2024-03-31T15:55:19.548666+0000","last_became_active":"2024-03-31T15:55:18.583246+0000","last_became_peered":"2024-03-31T15:55:18.583246+0000","last_unstale":"2024-03-31T15:55:19.548666+0000","last_undegraded":"2024-03-31T15:55:19.548666+0000","last_fullsized":"2024-03-31T15:55:19.548666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:48:46.491057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120947+0000","last_change":"2024-03-31T15:55:17.563636+0000","last_active":"2024-03-31T15:55:21.120947+0000","last_peered":"2024-03-31T15:55:21.120947+0000","last_clean":"2024-03-31T15:55:21.120947+0000","last_became_active":"2024-03-31T15:55:17.562976+0000","last_became_peered":"2024-03-31T15:55:17.562976+0000","last_unstale":"2024-03-31T15:55:21.120947+0000","last_undegraded":"2024-03-31T15:55:21.120947+0000","last_fullsized":"2024-03-31T15:55:21.120947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837772+0000","last_change":"2024-03-31T15:55:18.567428+0000","last_active":"2024-03-31T15:55:20.837772+0000","last_peered":"2024-03-31T15:55:20.837772+0000","last_clean":"2024-03-31T15:55:20.837772+0000","last_became_active":"2024-03-31T15:55:18.566876+0000","last_became_peered":"2024-03-31T15:55:18.566876+0000","last_unstale":"2024-03-31T15:55:20.837772+0000","last_undegraded":"2024-03-31T15:55:20.837772+0000","last_fullsized":"2024-03-31T15:55:20.837772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553620+0000","last_change":"2024-03-31T15:55:17.564638+0000","last_active":"2024-03-31T15:55:19.553620+0000","last_peered":"2024-03-31T15:55:19.553620+0000","last_clean":"2024-03-31T15:55:19.553620+0000","last_became_active":"2024-03-31T15:55:17.564347+0000","last_became_peered":"2024-03-31T15:55:17.564347+0000","last_unstale":"2024-03-31T15:55:19.553620+0000","last_undegraded":"2024-03-31T15:55:19.553620+0000","last_fullsized":"2024-03-31T15:55:19.553620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:17.020690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553651+0000","last_change":"2024-03-31T15:55:18.567191+0000","last_active":"2024-03-31T15:55:19.553651+0000","last_peered":"2024-03-31T15:55:19.553651+0000","last_clean":"2024-03-31T15:55:19.553651+0000","last_became_active":"2024-03-31T15:55:18.566900+0000","last_became_peered":"2024-03-31T15:55:18.566900+0000","last_unstale":"2024-03-31T15:55:19.553651+0000","last_undegraded":"2024-03-31T15:55:19.553651+0000","last_fullsized":"2024-03-31T15:55:19.553651+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:03.221135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.577287+0000","last_change":"2024-03-31T15:55:17.569775+0000","last_active":"2024-03-31T15:55:18.577287+0000","last_peered":"2024-03-31T15:55:18.577287+0000","last_clean":"2024-03-31T15:55:18.577287+0000","last_became_active":"2024-03-31T15:55:17.568127+0000","last_became_peered":"2024-03-31T15:55:17.568127+0000","last_unstale":"2024-03-31T15:55:18.577287+0000","last_undegraded":"2024-03-31T15:55:18.577287+0000","last_fullsized":"2024-03-31T15:55:18.577287+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:19:15.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121038+0000","last_change":"2024-03-31T15:55:18.573156+0000","last_active":"2024-03-31T15:55:21.121038+0000","last_peered":"2024-03-31T15:55:21.121038+0000","last_clean":"2024-03-31T15:55:21.121038+0000","last_became_active":"2024-03-31T15:55:18.572316+0000","last_became_peered":"2024-03-31T15:55:18.572316+0000","last_unstale":"2024-03-31T15:55:21.121038+0000","last_undegraded":"2024-03-31T15:55:21.121038+0000","last_fullsized":"2024-03-31T15:55:21.121038+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:17.564811+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:17.564461+0000","last_became_peered":"2024-03-31T15:55:17.564461+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548413+0000","last_change":"2024-03-31T15:55:18.572911+0000","last_active":"2024-03-31T15:55:19.548413+0000","last_peered":"2024-03-31T15:55:19.548413+0000","last_clean":"2024-03-31T15:55:19.548413+0000","last_became_active":"2024-03-31T15:55:18.572663+0000","last_became_peered":"2024-03-31T15:55:18.572663+0000","last_unstale":"2024-03-31T15:55:19.548413+0000","last_undegraded":"2024-03-31T15:55:19.548413+0000","last_fullsized":"2024-03-31T15:55:19.548413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120482+0000","last_change":"2024-03-31T15:55:17.558598+0000","last_active":"2024-03-31T15:55:21.120482+0000","last_peered":"2024-03-31T15:55:21.120482+0000","last_clean":"2024-03-31T15:55:21.120482+0000","last_became_active":"2024-03-31T15:55:17.557967+0000","last_became_peered":"2024-03-31T15:55:17.557967+0000","last_unstale":"2024-03-31T15:55:21.120482+0000","last_undegraded":"2024-03-31T15:55:21.120482+0000","last_fullsized":"2024-03-31T15:55:21.120482+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550725+0000","last_change":"2024-03-31T15:55:18.558312+0000","last_active":"2024-03-31T15:55:19.550725+0000","last_peered":"2024-03-31T15:55:19.550725+0000","last_clean":"2024-03-31T15:55:19.550725+0000","last_became_active":"2024-03-31T15:55:18.556133+0000","last_became_peered":"2024-03-31T15:55:18.556133+0000","last_unstale":"2024-03-31T15:55:19.550725+0000","last_undegraded":"2024-03-31T15:55:19.550725+0000","last_fullsized":"2024-03-31T15:55:19.550725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:51:25.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548747+0000","last_change":"2024-03-31T15:55:17.559251+0000","last_active":"2024-03-31T15:55:19.548747+0000","last_peered":"2024-03-31T15:55:19.548747+0000","last_clean":"2024-03-31T15:55:19.548747+0000","last_became_active":"2024-03-31T15:55:17.558977+0000","last_became_peered":"2024-03-31T15:55:17.558977+0000","last_unstale":"2024-03-31T15:55:19.548747+0000","last_undegraded":"2024-03-31T15:55:19.548747+0000","last_fullsized":"2024-03-31T15:55:19.548747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:23.098269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583255+0000","last_change":"2024-03-31T15:55:18.576163+0000","last_active":"2024-03-31T15:55:18.583255+0000","last_peered":"2024-03-31T15:55:18.583255+0000","last_clean":"2024-03-31T15:55:18.583255+0000","last_became_active":"2024-03-31T15:55:18.575491+0000","last_became_peered":"2024-03-31T15:55:18.575491+0000","last_unstale":"2024-03-31T15:55:18.583255+0000","last_undegraded":"2024-03-31T15:55:18.583255+0000","last_fullsized":"2024-03-31T15:55:18.583255+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:25.287778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553997+0000","last_change":"2024-03-31T15:55:17.559025+0000","last_active":"2024-03-31T15:55:19.553997+0000","last_peered":"2024-03-31T15:55:19.553997+0000","last_clean":"2024-03-31T15:55:19.553997+0000","last_became_active":"2024-03-31T15:55:17.558732+0000","last_became_peered":"2024-03-31T15:55:17.558732+0000","last_unstale":"2024-03-31T15:55:19.553997+0000","last_undegraded":"2024-03-31T15:55:19.553997+0000","last_fullsized":"2024-03-31T15:55:19.553997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:03.959563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838073+0000","last_change":"2024-03-31T15:55:18.559732+0000","last_active":"2024-03-31T15:55:20.838073+0000","last_peered":"2024-03-31T15:55:20.838073+0000","last_clean":"2024-03-31T15:55:20.838073+0000","last_became_active":"2024-03-31T15:55:18.559432+0000","last_became_peered":"2024-03-31T15:55:18.559432+0000","last_unstale":"2024-03-31T15:55:20.838073+0000","last_undegraded":"2024-03-31T15:55:20.838073+0000","last_fullsized":"2024-03-31T15:55:20.838073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:15.200810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:17.559993+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:17.559669+0000","last_became_peered":"2024-03-31T15:55:17.559669+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:24:33.510910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121078+0000","last_change":"2024-03-31T15:55:18.558993+0000","last_active":"2024-03-31T15:55:21.121078+0000","last_peered":"2024-03-31T15:55:21.121078+0000","last_clean":"2024-03-31T15:55:21.121078+0000","last_became_active":"2024-03-31T15:55:18.558670+0000","last_became_peered":"2024-03-31T15:55:18.558670+0000","last_unstale":"2024-03-31T15:55:21.121078+0000","last_undegraded":"2024-03-31T15:55:21.121078+0000","last_fullsized":"2024-03-31T15:55:21.121078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548239+0000","last_change":"2024-03-31T15:55:17.561703+0000","last_active":"2024-03-31T15:55:19.548239+0000","last_peered":"2024-03-31T15:55:19.548239+0000","last_clean":"2024-03-31T15:55:19.548239+0000","last_became_active":"2024-03-31T15:55:17.561448+0000","last_became_peered":"2024-03-31T15:55:17.561448+0000","last_unstale":"2024-03-31T15:55:19.548239+0000","last_undegraded":"2024-03-31T15:55:19.548239+0000","last_fullsized":"2024-03-31T15:55:19.548239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:38.684847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837750+0000","last_change":"2024-03-31T15:55:18.573704+0000","last_active":"2024-03-31T15:55:20.837750+0000","last_peered":"2024-03-31T15:55:20.837750+0000","last_clean":"2024-03-31T15:55:20.837750+0000","last_became_active":"2024-03-31T15:55:18.573462+0000","last_became_peered":"2024-03-31T15:55:18.573462+0000","last_unstale":"2024-03-31T15:55:20.837750+0000","last_undegraded":"2024-03-31T15:55:20.837750+0000","last_fullsized":"2024-03-31T15:55:20.837750+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548566+0000","last_change":"2024-03-31T15:55:17.565178+0000","last_active":"2024-03-31T15:55:19.548566+0000","last_peered":"2024-03-31T15:55:19.548566+0000","last_clean":"2024-03-31T15:55:19.548566+0000","last_became_active":"2024-03-31T15:55:17.564972+0000","last_became_peered":"2024-03-31T15:55:17.564972+0000","last_unstale":"2024-03-31T15:55:19.548566+0000","last_undegraded":"2024-03-31T15:55:19.548566+0000","last_fullsized":"2024-03-31T15:55:19.548566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:43.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548114+0000","last_change":"2024-03-31T15:55:18.576538+0000","last_active":"2024-03-31T15:55:19.548114+0000","last_peered":"2024-03-31T15:55:19.548114+0000","last_clean":"2024-03-31T15:55:19.548114+0000","last_became_active":"2024-03-31T15:55:18.569912+0000","last_became_peered":"2024-03-31T15:55:18.569912+0000","last_unstale":"2024-03-31T15:55:19.548114+0000","last_undegraded":"2024-03-31T15:55:19.548114+0000","last_fullsized":"2024-03-31T15:55:19.548114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930236+0000","last_change":"2024-03-31T15:55:17.564498+0000","last_active":"2024-03-31T15:55:20.930236+0000","last_peered":"2024-03-31T15:55:20.930236+0000","last_clean":"2024-03-31T15:55:20.930236+0000","last_became_active":"2024-03-31T15:55:17.564148+0000","last_became_peered":"2024-03-31T15:55:17.564148+0000","last_unstale":"2024-03-31T15:55:20.930236+0000","last_undegraded":"2024-03-31T15:55:20.930236+0000","last_fullsized":"2024-03-31T15:55:20.930236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:42:11.639287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.580350+0000","last_change":"2024-03-31T15:55:18.567828+0000","last_active":"2024-03-31T15:55:18.580350+0000","last_peered":"2024-03-31T15:55:18.580350+0000","last_clean":"2024-03-31T15:55:18.580350+0000","last_became_active":"2024-03-31T15:55:18.567508+0000","last_became_peered":"2024-03-31T15:55:18.567508+0000","last_unstale":"2024-03-31T15:55:18.580350+0000","last_undegraded":"2024-03-31T15:55:18.580350+0000","last_fullsized":"2024-03-31T15:55:18.580350+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:39.692689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547389+0000","last_change":"2024-03-31T15:55:17.568236+0000","last_active":"2024-03-31T15:55:19.547389+0000","last_peered":"2024-03-31T15:55:19.547389+0000","last_clean":"2024-03-31T15:55:19.547389+0000","last_became_active":"2024-03-31T15:55:17.567952+0000","last_became_peered":"2024-03-31T15:55:17.567952+0000","last_unstale":"2024-03-31T15:55:19.547389+0000","last_undegraded":"2024-03-31T15:55:19.547389+0000","last_fullsized":"2024-03-31T15:55:19.547389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548449+0000","last_change":"2024-03-31T15:55:18.570563+0000","last_active":"2024-03-31T15:55:19.548449+0000","last_peered":"2024-03-31T15:55:19.548449+0000","last_clean":"2024-03-31T15:55:19.548449+0000","last_became_active":"2024-03-31T15:55:18.570292+0000","last_became_peered":"2024-03-31T15:55:18.570292+0000","last_unstale":"2024-03-31T15:55:19.548449+0000","last_undegraded":"2024-03-31T15:55:19.548449+0000","last_fullsized":"2024-03-31T15:55:19.548449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.584994+0000","last_change":"2024-03-31T15:55:17.568544+0000","last_active":"2024-03-31T15:55:18.584994+0000","last_peered":"2024-03-31T15:55:18.584994+0000","last_clean":"2024-03-31T15:55:18.584994+0000","last_became_active":"2024-03-31T15:55:17.567942+0000","last_became_peered":"2024-03-31T15:55:17.567942+0000","last_unstale":"2024-03-31T15:55:18.584994+0000","last_undegraded":"2024-03-31T15:55:18.584994+0000","last_fullsized":"2024-03-31T15:55:18.584994+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:10.013322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548171+0000","last_change":"2024-03-31T15:55:18.574899+0000","last_active":"2024-03-31T15:55:19.548171+0000","last_peered":"2024-03-31T15:55:19.548171+0000","last_clean":"2024-03-31T15:55:19.548171+0000","last_became_active":"2024-03-31T15:55:18.574347+0000","last_became_peered":"2024-03-31T15:55:18.574347+0000","last_unstale":"2024-03-31T15:55:19.548171+0000","last_undegraded":"2024-03-31T15:55:19.548171+0000","last_fullsized":"2024-03-31T15:55:19.548171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554076+0000","last_change":"2024-03-31T15:55:17.564758+0000","last_active":"2024-03-31T15:55:19.554076+0000","last_peered":"2024-03-31T15:55:19.554076+0000","last_clean":"2024-03-31T15:55:19.554076+0000","last_became_active":"2024-03-31T15:55:17.564417+0000","last_became_peered":"2024-03-31T15:55:17.564417+0000","last_unstale":"2024-03-31T15:55:19.554076+0000","last_undegraded":"2024-03-31T15:55:19.554076+0000","last_fullsized":"2024-03-31T15:55:19.554076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:01.251935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:18.569677+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:18.569325+0000","last_became_peered":"2024-03-31T15:55:18.569325+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:50:35.297846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121071+0000","last_change":"2024-03-31T15:55:17.563289+0000","last_active":"2024-03-31T15:55:21.121071+0000","last_peered":"2024-03-31T15:55:21.121071+0000","last_clean":"2024-03-31T15:55:21.121071+0000","last_became_active":"2024-03-31T15:55:17.562945+0000","last_became_peered":"2024-03-31T15:55:17.562945+0000","last_unstale":"2024-03-31T15:55:21.121071+0000","last_undegraded":"2024-03-31T15:55:21.121071+0000","last_fullsized":"2024-03-31T15:55:21.121071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120928+0000","last_change":"2024-03-31T15:55:18.557599+0000","last_active":"2024-03-31T15:55:21.120928+0000","last_peered":"2024-03-31T15:55:21.120928+0000","last_clean":"2024-03-31T15:55:21.120928+0000","last_became_active":"2024-03-31T15:55:18.557275+0000","last_became_peered":"2024-03-31T15:55:18.557275+0000","last_unstale":"2024-03-31T15:55:21.120928+0000","last_undegraded":"2024-03-31T15:55:21.120928+0000","last_fullsized":"2024-03-31T15:55:21.120928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550772+0000","last_change":"2024-03-31T15:55:17.562464+0000","last_active":"2024-03-31T15:55:19.550772+0000","last_peered":"2024-03-31T15:55:19.550772+0000","last_clean":"2024-03-31T15:55:19.550772+0000","last_became_active":"2024-03-31T15:55:17.562192+0000","last_became_peered":"2024-03-31T15:55:17.562192+0000","last_unstale":"2024-03-31T15:55:19.550772+0000","last_undegraded":"2024-03-31T15:55:19.550772+0000","last_fullsized":"2024-03-31T15:55:19.550772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:13.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550931+0000","last_change":"2024-03-31T15:55:18.572667+0000","last_active":"2024-03-31T15:55:19.550931+0000","last_peered":"2024-03-31T15:55:19.550931+0000","last_clean":"2024-03-31T15:55:19.550931+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.550931+0000","last_undegraded":"2024-03-31T15:55:19.550931+0000","last_fullsized":"2024-03-31T15:55:19.550931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:54:13.587576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930339+0000","last_change":"2024-03-31T15:55:17.564482+0000","last_active":"2024-03-31T15:55:20.930339+0000","last_peered":"2024-03-31T15:55:20.930339+0000","last_clean":"2024-03-31T15:55:20.930339+0000","last_became_active":"2024-03-31T15:55:17.564263+0000","last_became_peered":"2024-03-31T15:55:17.564263+0000","last_unstale":"2024-03-31T15:55:20.930339+0000","last_undegraded":"2024-03-31T15:55:20.930339+0000","last_fullsized":"2024-03-31T15:55:20.930339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583198+0000","last_change":"2024-03-31T15:55:18.573739+0000","last_active":"2024-03-31T15:55:18.583198+0000","last_peered":"2024-03-31T15:55:18.583198+0000","last_clean":"2024-03-31T15:55:18.583198+0000","last_became_active":"2024-03-31T15:55:18.573288+0000","last_became_peered":"2024-03-31T15:55:18.573288+0000","last_unstale":"2024-03-31T15:55:18.583198+0000","last_undegraded":"2024-03-31T15:55:18.583198+0000","last_fullsized":"2024-03-31T15:55:18.583198+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:27.032794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550590+0000","last_change":"2024-03-31T15:55:17.561760+0000","last_active":"2024-03-31T15:55:19.550590+0000","last_peered":"2024-03-31T15:55:19.550590+0000","last_clean":"2024-03-31T15:55:19.550590+0000","last_became_active":"2024-03-31T15:55:17.561471+0000","last_became_peered":"2024-03-31T15:55:17.561471+0000","last_unstale":"2024-03-31T15:55:19.550590+0000","last_undegraded":"2024-03-31T15:55:19.550590+0000","last_fullsized":"2024-03-31T15:55:19.550590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:31:27.557663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837752+0000","last_change":"2024-03-31T15:55:18.557363+0000","last_active":"2024-03-31T15:55:20.837752+0000","last_peered":"2024-03-31T15:55:20.837752+0000","last_clean":"2024-03-31T15:55:20.837752+0000","last_became_active":"2024-03-31T15:55:18.557133+0000","last_became_peered":"2024-03-31T15:55:18.557133+0000","last_unstale":"2024-03-31T15:55:20.837752+0000","last_undegraded":"2024-03-31T15:55:20.837752+0000","last_fullsized":"2024-03-31T15:55:20.837752+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550455+0000","last_change":"2024-03-31T15:55:17.567522+0000","last_active":"2024-03-31T15:55:19.550455+0000","last_peered":"2024-03-31T15:55:19.550455+0000","last_clean":"2024-03-31T15:55:19.550455+0000","last_became_active":"2024-03-31T15:55:17.567296+0000","last_became_peered":"2024-03-31T15:55:17.567296+0000","last_unstale":"2024-03-31T15:55:19.550455+0000","last_undegraded":"2024-03-31T15:55:19.550455+0000","last_fullsized":"2024-03-31T15:55:19.550455+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:24:18.305487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553568+0000","last_change":"2024-03-31T15:55:18.578437+0000","last_active":"2024-03-31T15:55:19.553568+0000","last_peered":"2024-03-31T15:55:19.553568+0000","last_clean":"2024-03-31T15:55:19.553568+0000","last_became_active":"2024-03-31T15:55:18.578122+0000","last_became_peered":"2024-03-31T15:55:18.578122+0000","last_unstale":"2024-03-31T15:55:19.553568+0000","last_undegraded":"2024-03-31T15:55:19.553568+0000","last_fullsized":"2024-03-31T15:55:19.553568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:24.502072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554616+0000","last_change":"2024-03-31T15:55:17.569573+0000","last_active":"2024-03-31T15:55:18.554616+0000","last_peered":"2024-03-31T15:55:18.554616+0000","last_clean":"2024-03-31T15:55:18.554616+0000","last_became_active":"2024-03-31T15:55:17.569295+0000","last_became_peered":"2024-03-31T15:55:17.569295+0000","last_unstale":"2024-03-31T15:55:18.554616+0000","last_undegraded":"2024-03-31T15:55:18.554616+0000","last_fullsized":"2024-03-31T15:55:18.554616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:11.923571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548574+0000","last_change":"2024-03-31T15:55:18.581151+0000","last_active":"2024-03-31T15:55:19.548574+0000","last_peered":"2024-03-31T15:55:19.548574+0000","last_clean":"2024-03-31T15:55:19.548574+0000","last_became_active":"2024-03-31T15:55:18.580843+0000","last_became_peered":"2024-03-31T15:55:18.580843+0000","last_unstale":"2024-03-31T15:55:19.548574+0000","last_undegraded":"2024-03-31T15:55:19.548574+0000","last_fullsized":"2024-03-31T15:55:19.548574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550481+0000","last_change":"2024-03-31T15:55:17.561569+0000","last_active":"2024-03-31T15:55:19.550481+0000","last_peered":"2024-03-31T15:55:19.550481+0000","last_clean":"2024-03-31T15:55:19.550481+0000","last_became_active":"2024-03-31T15:55:17.560629+0000","last_became_peered":"2024-03-31T15:55:17.560629+0000","last_unstale":"2024-03-31T15:55:19.550481+0000","last_undegraded":"2024-03-31T15:55:19.550481+0000","last_fullsized":"2024-03-31T15:55:19.550481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:36.826694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930424+0000","last_change":"2024-03-31T15:55:18.569951+0000","last_active":"2024-03-31T15:55:20.930424+0000","last_peered":"2024-03-31T15:55:20.930424+0000","last_clean":"2024-03-31T15:55:20.930424+0000","last_became_active":"2024-03-31T15:55:18.569622+0000","last_became_peered":"2024-03-31T15:55:18.569622+0000","last_unstale":"2024-03-31T15:55:20.930424+0000","last_undegraded":"2024-03-31T15:55:20.930424+0000","last_fullsized":"2024-03-31T15:55:20.930424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554129+0000","last_change":"2024-03-31T15:55:17.558967+0000","last_active":"2024-03-31T15:55:19.554129+0000","last_peered":"2024-03-31T15:55:19.554129+0000","last_clean":"2024-03-31T15:55:19.554129+0000","last_became_active":"2024-03-31T15:55:17.558642+0000","last_became_peered":"2024-03-31T15:55:17.558642+0000","last_unstale":"2024-03-31T15:55:19.554129+0000","last_undegraded":"2024-03-31T15:55:19.554129+0000","last_fullsized":"2024-03-31T15:55:19.554129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:47.530373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583276+0000","last_change":"2024-03-31T15:55:18.576406+0000","last_active":"2024-03-31T15:55:18.583276+0000","last_peered":"2024-03-31T15:55:18.583276+0000","last_clean":"2024-03-31T15:55:18.583276+0000","last_became_active":"2024-03-31T15:55:18.575892+0000","last_became_peered":"2024-03-31T15:55:18.575892+0000","last_unstale":"2024-03-31T15:55:18.583276+0000","last_undegraded":"2024-03-31T15:55:18.583276+0000","last_fullsized":"2024-03-31T15:55:18.583276+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:01:01.439448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.552589+0000","last_change":"2024-03-31T15:55:17.565054+0000","last_active":"2024-03-31T15:55:19.552589+0000","last_peered":"2024-03-31T15:55:19.552589+0000","last_clean":"2024-03-31T15:55:19.552589+0000","last_became_active":"2024-03-31T15:55:17.564845+0000","last_became_peered":"2024-03-31T15:55:17.564845+0000","last_unstale":"2024-03-31T15:55:19.552589+0000","last_undegraded":"2024-03-31T15:55:19.552589+0000","last_fullsized":"2024-03-31T15:55:19.552589+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:34:03.576042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548610+0000","last_change":"2024-03-31T15:55:18.572781+0000","last_active":"2024-03-31T15:55:19.548610+0000","last_peered":"2024-03-31T15:55:19.548610+0000","last_clean":"2024-03-31T15:55:19.548610+0000","last_became_active":"2024-03-31T15:55:18.572475+0000","last_became_peered":"2024-03-31T15:55:18.572475+0000","last_unstale":"2024-03-31T15:55:19.548610+0000","last_undegraded":"2024-03-31T15:55:19.548610+0000","last_fullsized":"2024-03-31T15:55:19.548610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:21.840147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838034+0000","last_change":"2024-03-31T15:55:17.552434+0000","last_active":"2024-03-31T15:55:20.838034+0000","last_peered":"2024-03-31T15:55:20.838034+0000","last_clean":"2024-03-31T15:55:20.838034+0000","last_became_active":"2024-03-31T15:55:17.552137+0000","last_became_peered":"2024-03-31T15:55:17.552137+0000","last_unstale":"2024-03-31T15:55:20.838034+0000","last_undegraded":"2024-03-31T15:55:20.838034+0000","last_fullsized":"2024-03-31T15:55:20.838034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837915+0000","last_change":"2024-03-31T15:55:18.567217+0000","last_active":"2024-03-31T15:55:20.837915+0000","last_peered":"2024-03-31T15:55:20.837915+0000","last_clean":"2024-03-31T15:55:20.837915+0000","last_became_active":"2024-03-31T15:55:18.566527+0000","last_became_peered":"2024-03-31T15:55:18.566527+0000","last_unstale":"2024-03-31T15:55:20.837915+0000","last_undegraded":"2024-03-31T15:55:20.837915+0000","last_fullsized":"2024-03-31T15:55:20.837915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837837+0000","last_change":"2024-03-31T15:55:17.569970+0000","last_active":"2024-03-31T15:55:20.837837+0000","last_peered":"2024-03-31T15:55:20.837837+0000","last_clean":"2024-03-31T15:55:20.837837+0000","last_became_active":"2024-03-31T15:55:17.569178+0000","last_became_peered":"2024-03-31T15:55:17.569178+0000","last_unstale":"2024-03-31T15:55:20.837837+0000","last_undegraded":"2024-03-31T15:55:20.837837+0000","last_fullsized":"2024-03-31T15:55:20.837837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:18.576213+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:18.575930+0000","last_became_peered":"2024-03-31T15:55:18.575930+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547246+0000","last_change":"2024-03-31T15:55:17.569351+0000","last_active":"2024-03-31T15:55:19.547246+0000","last_peered":"2024-03-31T15:55:19.547246+0000","last_clean":"2024-03-31T15:55:19.547246+0000","last_became_active":"2024-03-31T15:55:17.569079+0000","last_became_peered":"2024-03-31T15:55:17.569079+0000","last_unstale":"2024-03-31T15:55:19.547246+0000","last_undegraded":"2024-03-31T15:55:19.547246+0000","last_fullsized":"2024-03-31T15:55:19.547246+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120705+0000","last_change":"2024-03-31T15:55:18.572994+0000","last_active":"2024-03-31T15:55:21.120705+0000","last_peered":"2024-03-31T15:55:21.120705+0000","last_clean":"2024-03-31T15:55:21.120705+0000","last_became_active":"2024-03-31T15:55:18.572692+0000","last_became_peered":"2024-03-31T15:55:18.572692+0000","last_unstale":"2024-03-31T15:55:21.120705+0000","last_undegraded":"2024-03-31T15:55:21.120705+0000","last_fullsized":"2024-03-31T15:55:21.120705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120545+0000","last_change":"2024-03-31T15:55:17.563312+0000","last_active":"2024-03-31T15:55:21.120545+0000","last_peered":"2024-03-31T15:55:21.120545+0000","last_clean":"2024-03-31T15:55:21.120545+0000","last_became_active":"2024-03-31T15:55:17.562982+0000","last_became_peered":"2024-03-31T15:55:17.562982+0000","last_unstale":"2024-03-31T15:55:21.120545+0000","last_undegraded":"2024-03-31T15:55:21.120545+0000","last_fullsized":"2024-03-31T15:55:21.120545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.583180+0000","last_change":"2024-03-31T15:55:18.575859+0000","last_active":"2024-03-31T15:55:18.583180+0000","last_peered":"2024-03-31T15:55:18.583180+0000","last_clean":"2024-03-31T15:55:18.583180+0000","last_became_active":"2024-03-31T15:55:18.575507+0000","last_became_peered":"2024-03-31T15:55:18.575507+0000","last_unstale":"2024-03-31T15:55:18.583180+0000","last_undegraded":"2024-03-31T15:55:18.583180+0000","last_fullsized":"2024-03-31T15:55:18.583180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:31:28.813601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547913+0000","last_change":"2024-03-31T15:55:17.559862+0000","last_active":"2024-03-31T15:55:19.547913+0000","last_peered":"2024-03-31T15:55:19.547913+0000","last_clean":"2024-03-31T15:55:19.547913+0000","last_became_active":"2024-03-31T15:55:17.558616+0000","last_became_peered":"2024-03-31T15:55:17.558616+0000","last_unstale":"2024-03-31T15:55:19.547913+0000","last_undegraded":"2024-03-31T15:55:19.547913+0000","last_fullsized":"2024-03-31T15:55:19.547913+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:28.724128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553996+0000","last_change":"2024-03-31T15:55:18.582994+0000","last_active":"2024-03-31T15:55:19.553996+0000","last_peered":"2024-03-31T15:55:19.553996+0000","last_clean":"2024-03-31T15:55:19.553996+0000","last_became_active":"2024-03-31T15:55:18.580575+0000","last_became_peered":"2024-03-31T15:55:18.580575+0000","last_unstale":"2024-03-31T15:55:19.553996+0000","last_undegraded":"2024-03-31T15:55:19.553996+0000","last_fullsized":"2024-03-31T15:55:19.553996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:59.330574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.554197+0000","last_change":"2024-03-31T15:55:17.570487+0000","last_active":"2024-03-31T15:55:18.554197+0000","last_peered":"2024-03-31T15:55:18.554197+0000","last_clean":"2024-03-31T15:55:18.554197+0000","last_became_active":"2024-03-31T15:55:17.570204+0000","last_became_peered":"2024-03-31T15:55:17.570204+0000","last_unstale":"2024-03-31T15:55:18.554197+0000","last_undegraded":"2024-03-31T15:55:18.554197+0000","last_fullsized":"2024-03-31T15:55:18.554197+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:34.516949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-03-31T15:55:18.553800+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:55:18.553800+0000","last_peered":"2024-03-31T15:55:18.553800+0000","last_clean":"2024-03-31T15:55:18.553800+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:55:18.553800+0000","last_undegraded":"2024-03-31T15:55:18.553800+0000","last_fullsized":"2024-03-31T15:55:18.553800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509464,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":192513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509464,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":450560,"data_stored":192043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509465,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":434176,"data_stored":185329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":782333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":186885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542168,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":185309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1040384,"data_stored":777289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:55:27.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:55:27.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 Processor -- start 2024-03-31T15:55:27.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 -- start start 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0e1c15c2e0 0x7f0e1c073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0e1c0b0db0 0x7f0e1c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e1c0b59a0 con 0x7f0e1c0b5590 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0e1c15c6c0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e21200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0e1c0b1b20 con 0x7f0e1c0b0db0 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0e1c15c2e0 0x7f0e1c073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0e1c0b5590 0x7f0e1c0ad1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44306/0 (socket says 172.21.15.90:44306) 2024-03-31T15:55:27.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1be00640 1 -- 172.21.15.90:0/434025878 learned_addr learned my addr 172.21.15.90:0/434025878 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e13e00640 1 --2- 172.21.15.90:0/434025878 >> v2:172.21.15.102:3300/0 conn(0x7f0e1c0b0db0 0x7f0e1c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1aa00640 1 -- 172.21.15.90:0/434025878 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699114729 0 0) 0x7f0e140014a0 con 0x7f0e1c0b5590 2024-03-31T15:55:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1aa00640 1 -- 172.21.15.90:0/434025878 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e00003530 con 0x7f0e1c0b5590 2024-03-31T15:55:27.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1b400640 1 -- 172.21.15.90:0/434025878 >> v2:172.21.15.102:3300/0 conn(0x7f0e1c0b0db0 msgr2=0x7f0e1c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1b400640 1 --2- 172.21.15.90:0/434025878 >> v2:172.21.15.102:3300/0 conn(0x7f0e1c0b0db0 0x7f0e1c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1b400640 1 -- 172.21.15.90:0/434025878 >> v1:172.21.15.90:6789/0 conn(0x7f0e1c0b5590 legacy=0x7f0e1c0ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.444+0000 7f0e1b400640 1 -- 172.21.15.90:0/434025878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1c0b02d0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e1b400640 1 --2- 172.21.15.90:0/434025878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c073870 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0ccca0 tx=0x7f0e0c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31a0ac6790e6bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e1aa00640 1 -- 172.21.15.90:0/434025878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c0ab650 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e1aa00640 1 -- 172.21.15.90:0/434025878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e0c0b0070 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e1aa00640 1 -- 172.21.15.90:0/434025878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c0aba10 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 -- 172.21.15.90:0/434025878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 msgr2=0x7f0e1c073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 --2- 172.21.15.90:0/434025878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c073870 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0ccca0 tx=0x7f0e0c01c450 comp rx=0 tx=0).stop 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 -- 172.21.15.90:0/434025878 shutdown_connections 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 --2- 172.21.15.90:0/434025878 >> v2:172.21.15.102:3300/0 conn(0x7f0e1c0b0db0 0x7f0e1c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 --2- 172.21.15.90:0/434025878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c073870 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 -- 172.21.15.90:0/434025878 >> 172.21.15.90:0/434025878 conn(0x7f0e1c072d80 msgr2=0x7f0e1c073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 -- 172.21.15.90:0/434025878 shutdown_connections 2024-03-31T15:55:27.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.448+0000 7f0e21200640 1 -- 172.21.15.90:0/434025878 wait complete. 2024-03-31T15:55:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 Processor -- start 2024-03-31T15:55:27.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- start start 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0e1c0b0db0 0x7f0e1c099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e1c0b5590 0x7f0e1c0ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c0b1340 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1c0b11c0 con 0x7f0e1c0b5590 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0e1c0afef0 con 0x7f0e1c0b0db0 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e13e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e1c0b5590 0x7f0e1c0ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50904/0 (socket says 172.21.15.90:50904) 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0e1c0b0db0 0x7f0e1c099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 -- 172.21.15.90:0/2309647956 learned_addr learned my addr 172.21.15.90:0/2309647956 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 -- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0e1c0b0db0 msgr2=0x7f0e1c099700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0e1c0b0db0 0x7f0e1c099700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 -- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e1c0b5590 msgr2=0x7f0e1c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e1c0b5590 0x7f0e1c0ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 -- 172.21.15.90:0/2309647956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e1c099ce0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e1be00640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400cb80 tx=0x7f0e1400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14017070 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1400ad40 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14012620 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1c0c9100 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e1c0c96c0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.452+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e1c068d00 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.456+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0e1401a070 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.456+0000 7f0e13400640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 0x7f0df806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.456+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0e140869d0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.456+0000 7f0e1b400640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 0x7f0df806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.456+0000 7f0e1b400640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 0x7f0df806a840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0af000 tx=0x7f0e0c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.460+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0e1c0b59a0 con 0x7f0e1c15c2e0 2024-03-31T15:55:27.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.616+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e1c0ce9b0 con 0x7f0df8068380 2024-03-31T15:55:27.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.624+0000 7f0e13400640 1 -- 172.21.15.90:0/2309647956 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251871 (secure 0 0 0) 0x7f0e1c0ce9b0 con 0x7f0df8068380 2024-03-31T15:55:27.619 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:27.620 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.624+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 msgr2=0x7f0df806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.624+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 0x7f0df806a840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0af000 tx=0x7f0e0c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.624+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 msgr2=0x7f0e1c0c4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.624+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400cb80 tx=0x7f0e1400a430 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 shutdown_connections 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0df8068380 0x7f0df806a840 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0e1c15c2e0 0x7f0e1c0c4f00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e1c0b5590 0x7f0e1c0ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 --2- 172.21.15.90:0/2309647956 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0e1c0b0db0 0x7f0e1c099700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 >> 172.21.15.90:0/2309647956 conn(0x7f0e1c072d80 msgr2=0x7f0e1c1597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:27.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 shutdown_connections 2024-03-31T15:55:27.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.628+0000 7f0e21200640 1 -- 172.21.15.90:0/2309647956 wait complete. 2024-03-31T15:55:27.640 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-03-31T15:55:27.239948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220872,"kb_used_data":4680,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753848,"statfs":{"total":773094113280,"available":772867940352,"internally_reserved":0,"allocated":4792320,"data_stored":2721004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.696061"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548723+0000","last_change":"2024-03-31T15:55:18.569836+0000","last_active":"2024-03-31T15:55:19.548723+0000","last_peered":"2024-03-31T15:55:19.548723+0000","last_clean":"2024-03-31T15:55:19.548723+0000","last_became_active":"2024-03-31T15:55:18.569542+0000","last_became_peered":"2024-03-31T15:55:18.569542+0000","last_unstale":"2024-03-31T15:55:19.548723+0000","last_undegraded":"2024-03-31T15:55:19.548723+0000","last_fullsized":"2024-03-31T15:55:19.548723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:02:52.818181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548259+0000","last_change":"2024-03-31T15:55:18.570636+0000","last_active":"2024-03-31T15:55:19.548259+0000","last_peered":"2024-03-31T15:55:19.548259+0000","last_clean":"2024-03-31T15:55:19.548259+0000","last_became_active":"2024-03-31T15:55:18.570347+0000","last_became_peered":"2024-03-31T15:55:18.570347+0000","last_unstale":"2024-03-31T15:55:19.548259+0000","last_undegraded":"2024-03-31T15:55:19.548259+0000","last_fullsized":"2024-03-31T15:55:19.548259+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838065+0000","last_change":"2024-03-31T15:55:18.576959+0000","last_active":"2024-03-31T15:55:20.838065+0000","last_peered":"2024-03-31T15:55:20.838065+0000","last_clean":"2024-03-31T15:55:20.838065+0000","last_became_active":"2024-03-31T15:55:18.576703+0000","last_became_peered":"2024-03-31T15:55:18.576703+0000","last_unstale":"2024-03-31T15:55:20.838065+0000","last_undegraded":"2024-03-31T15:55:20.838065+0000","last_fullsized":"2024-03-31T15:55:20.838065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550774+0000","last_change":"2024-03-31T15:55:18.558889+0000","last_active":"2024-03-31T15:55:19.550774+0000","last_peered":"2024-03-31T15:55:19.550774+0000","last_clean":"2024-03-31T15:55:19.550774+0000","last_became_active":"2024-03-31T15:55:18.558620+0000","last_became_peered":"2024-03-31T15:55:18.558620+0000","last_unstale":"2024-03-31T15:55:19.550774+0000","last_undegraded":"2024-03-31T15:55:19.550774+0000","last_fullsized":"2024-03-31T15:55:19.550774+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:02:16.723939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550456+0000","last_change":"2024-03-31T15:55:18.552183+0000","last_active":"2024-03-31T15:55:19.550456+0000","last_peered":"2024-03-31T15:55:19.550456+0000","last_clean":"2024-03-31T15:55:19.550456+0000","last_became_active":"2024-03-31T15:55:18.551886+0000","last_became_peered":"2024-03-31T15:55:18.551886+0000","last_unstale":"2024-03-31T15:55:19.550456+0000","last_undegraded":"2024-03-31T15:55:19.550456+0000","last_fullsized":"2024-03-31T15:55:19.550456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:53:07.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930104+0000","last_change":"2024-03-31T15:55:18.577900+0000","last_active":"2024-03-31T15:55:20.930104+0000","last_peered":"2024-03-31T15:55:20.930104+0000","last_clean":"2024-03-31T15:55:20.930104+0000","last_became_active":"2024-03-31T15:55:18.577638+0000","last_became_peered":"2024-03-31T15:55:18.577638+0000","last_unstale":"2024-03-31T15:55:20.930104+0000","last_undegraded":"2024-03-31T15:55:20.930104+0000","last_fullsized":"2024-03-31T15:55:20.930104+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553456+0000","last_change":"2024-03-31T15:55:18.569983+0000","last_active":"2024-03-31T15:55:19.553456+0000","last_peered":"2024-03-31T15:55:19.553456+0000","last_clean":"2024-03-31T15:55:19.553456+0000","last_became_active":"2024-03-31T15:55:18.569706+0000","last_became_peered":"2024-03-31T15:55:18.569706+0000","last_unstale":"2024-03-31T15:55:19.553456+0000","last_undegraded":"2024-03-31T15:55:19.553456+0000","last_fullsized":"2024-03-31T15:55:19.553456+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:17:54.427505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547884+0000","last_change":"2024-03-31T15:55:18.572173+0000","last_active":"2024-03-31T15:55:19.547884+0000","last_peered":"2024-03-31T15:55:19.547884+0000","last_clean":"2024-03-31T15:55:19.547884+0000","last_became_active":"2024-03-31T15:55:18.571909+0000","last_became_peered":"2024-03-31T15:55:18.571909+0000","last_unstale":"2024-03-31T15:55:19.547884+0000","last_undegraded":"2024-03-31T15:55:19.547884+0000","last_fullsized":"2024-03-31T15:55:19.547884+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838187+0000","last_change":"2024-03-31T15:55:18.577912+0000","last_active":"2024-03-31T15:55:20.838187+0000","last_peered":"2024-03-31T15:55:20.838187+0000","last_clean":"2024-03-31T15:55:20.838187+0000","last_became_active":"2024-03-31T15:55:18.574214+0000","last_became_peered":"2024-03-31T15:55:18.574214+0000","last_unstale":"2024-03-31T15:55:20.838187+0000","last_undegraded":"2024-03-31T15:55:20.838187+0000","last_fullsized":"2024-03-31T15:55:20.838187+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:47:48.143140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553957+0000","last_change":"2024-03-31T15:55:18.567327+0000","last_active":"2024-03-31T15:55:19.553957+0000","last_peered":"2024-03-31T15:55:19.553957+0000","last_clean":"2024-03-31T15:55:19.553957+0000","last_became_active":"2024-03-31T15:55:18.567046+0000","last_became_peered":"2024-03-31T15:55:18.567046+0000","last_unstale":"2024-03-31T15:55:19.553957+0000","last_undegraded":"2024-03-31T15:55:19.553957+0000","last_fullsized":"2024-03-31T15:55:19.553957+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:21.180135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553799+0000","last_change":"2024-03-31T15:55:18.562702+0000","last_active":"2024-03-31T15:55:19.553799+0000","last_peered":"2024-03-31T15:55:19.553799+0000","last_clean":"2024-03-31T15:55:19.553799+0000","last_became_active":"2024-03-31T15:55:18.562447+0000","last_became_peered":"2024-03-31T15:55:18.562447+0000","last_unstale":"2024-03-31T15:55:19.553799+0000","last_undegraded":"2024-03-31T15:55:19.553799+0000","last_fullsized":"2024-03-31T15:55:19.553799+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:59:15.558471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838024+0000","last_change":"2024-03-31T15:55:18.562485+0000","last_active":"2024-03-31T15:55:20.838024+0000","last_peered":"2024-03-31T15:55:20.838024+0000","last_clean":"2024-03-31T15:55:20.838024+0000","last_became_active":"2024-03-31T15:55:18.562260+0000","last_became_peered":"2024-03-31T15:55:18.562260+0000","last_unstale":"2024-03-31T15:55:20.838024+0000","last_undegraded":"2024-03-31T15:55:20.838024+0000","last_fullsized":"2024-03-31T15:55:20.838024+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553436+0000","last_change":"2024-03-31T15:55:18.564320+0000","last_active":"2024-03-31T15:55:19.553436+0000","last_peered":"2024-03-31T15:55:19.553436+0000","last_clean":"2024-03-31T15:55:19.553436+0000","last_became_active":"2024-03-31T15:55:18.564064+0000","last_became_peered":"2024-03-31T15:55:18.564064+0000","last_unstale":"2024-03-31T15:55:19.553436+0000","last_undegraded":"2024-03-31T15:55:19.553436+0000","last_fullsized":"2024-03-31T15:55:19.553436+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:06.438509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120606+0000","last_change":"2024-03-31T15:55:18.581725+0000","last_active":"2024-03-31T15:55:21.120606+0000","last_peered":"2024-03-31T15:55:21.120606+0000","last_clean":"2024-03-31T15:55:21.120606+0000","last_became_active":"2024-03-31T15:55:18.581115+0000","last_became_peered":"2024-03-31T15:55:18.581115+0000","last_unstale":"2024-03-31T15:55:21.120606+0000","last_undegraded":"2024-03-31T15:55:21.120606+0000","last_fullsized":"2024-03-31T15:55:21.120606+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553656+0000","last_change":"2024-03-31T15:55:18.562134+0000","last_active":"2024-03-31T15:55:19.553656+0000","last_peered":"2024-03-31T15:55:19.553656+0000","last_clean":"2024-03-31T15:55:19.553656+0000","last_became_active":"2024-03-31T15:55:18.559688+0000","last_became_peered":"2024-03-31T15:55:18.559688+0000","last_unstale":"2024-03-31T15:55:19.553656+0000","last_undegraded":"2024-03-31T15:55:19.553656+0000","last_fullsized":"2024-03-31T15:55:19.553656+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:53:57.153194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547060+0000","last_change":"2024-03-31T15:55:18.573440+0000","last_active":"2024-03-31T15:55:19.547060+0000","last_peered":"2024-03-31T15:55:19.547060+0000","last_clean":"2024-03-31T15:55:19.547060+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.547060+0000","last_undegraded":"2024-03-31T15:55:19.547060+0000","last_fullsized":"2024-03-31T15:55:19.547060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:44:36.442599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548375+0000","last_change":"2024-03-31T15:55:18.571940+0000","last_active":"2024-03-31T15:55:19.548375+0000","last_peered":"2024-03-31T15:55:19.548375+0000","last_clean":"2024-03-31T15:55:19.548375+0000","last_became_active":"2024-03-31T15:55:18.571687+0000","last_became_peered":"2024-03-31T15:55:18.571687+0000","last_unstale":"2024-03-31T15:55:19.548375+0000","last_undegraded":"2024-03-31T15:55:19.548375+0000","last_fullsized":"2024-03-31T15:55:19.548375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547503+0000","last_change":"2024-03-31T15:55:18.581100+0000","last_active":"2024-03-31T15:55:19.547503+0000","last_peered":"2024-03-31T15:55:19.547503+0000","last_clean":"2024-03-31T15:55:19.547503+0000","last_became_active":"2024-03-31T15:55:18.580799+0000","last_became_peered":"2024-03-31T15:55:18.580799+0000","last_unstale":"2024-03-31T15:55:19.547503+0000","last_undegraded":"2024-03-31T15:55:19.547503+0000","last_fullsized":"2024-03-31T15:55:19.547503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:18.028532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550838+0000","last_change":"2024-03-31T15:55:18.565501+0000","last_active":"2024-03-31T15:55:19.550838+0000","last_peered":"2024-03-31T15:55:19.550838+0000","last_clean":"2024-03-31T15:55:19.550838+0000","last_became_active":"2024-03-31T15:55:18.565247+0000","last_became_peered":"2024-03-31T15:55:18.565247+0000","last_unstale":"2024-03-31T15:55:19.550838+0000","last_undegraded":"2024-03-31T15:55:19.550838+0000","last_fullsized":"2024-03-31T15:55:19.550838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:55.707922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:18.576219+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:18.575905+0000","last_became_peered":"2024-03-31T15:55:18.575905+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553636+0000","last_change":"2024-03-31T15:55:18.559315+0000","last_active":"2024-03-31T15:55:19.553636+0000","last_peered":"2024-03-31T15:55:19.553636+0000","last_clean":"2024-03-31T15:55:19.553636+0000","last_became_active":"2024-03-31T15:55:18.558870+0000","last_became_peered":"2024-03-31T15:55:18.558870+0000","last_unstale":"2024-03-31T15:55:19.553636+0000","last_undegraded":"2024-03-31T15:55:19.553636+0000","last_fullsized":"2024-03-31T15:55:19.553636+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:45.294220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121007+0000","last_change":"2024-03-31T15:55:18.571748+0000","last_active":"2024-03-31T15:55:21.121007+0000","last_peered":"2024-03-31T15:55:21.121007+0000","last_clean":"2024-03-31T15:55:21.121007+0000","last_became_active":"2024-03-31T15:55:18.571445+0000","last_became_peered":"2024-03-31T15:55:18.571445+0000","last_unstale":"2024-03-31T15:55:21.121007+0000","last_undegraded":"2024-03-31T15:55:21.121007+0000","last_fullsized":"2024-03-31T15:55:21.121007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547765+0000","last_change":"2024-03-31T15:55:18.575290+0000","last_active":"2024-03-31T15:55:19.547765+0000","last_peered":"2024-03-31T15:55:19.547765+0000","last_clean":"2024-03-31T15:55:19.547765+0000","last_became_active":"2024-03-31T15:55:18.574928+0000","last_became_peered":"2024-03-31T15:55:18.574928+0000","last_unstale":"2024-03-31T15:55:19.547765+0000","last_undegraded":"2024-03-31T15:55:19.547765+0000","last_fullsized":"2024-03-31T15:55:19.547765+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:29:10.150136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550628+0000","last_change":"2024-03-31T15:55:18.563439+0000","last_active":"2024-03-31T15:55:19.550628+0000","last_peered":"2024-03-31T15:55:19.550628+0000","last_clean":"2024-03-31T15:55:19.550628+0000","last_became_active":"2024-03-31T15:55:18.563160+0000","last_became_peered":"2024-03-31T15:55:18.563160+0000","last_unstale":"2024-03-31T15:55:19.550628+0000","last_undegraded":"2024-03-31T15:55:19.550628+0000","last_fullsized":"2024-03-31T15:55:19.550628+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:12:33.025431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547597+0000","last_change":"2024-03-31T15:55:18.582358+0000","last_active":"2024-03-31T15:55:19.547597+0000","last_peered":"2024-03-31T15:55:19.547597+0000","last_clean":"2024-03-31T15:55:19.547597+0000","last_became_active":"2024-03-31T15:55:18.581841+0000","last_became_peered":"2024-03-31T15:55:18.581841+0000","last_unstale":"2024-03-31T15:55:19.547597+0000","last_undegraded":"2024-03-31T15:55:19.547597+0000","last_fullsized":"2024-03-31T15:55:19.547597+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930482+0000","last_change":"2024-03-31T15:55:18.570686+0000","last_active":"2024-03-31T15:55:20.930482+0000","last_peered":"2024-03-31T15:55:20.930482+0000","last_clean":"2024-03-31T15:55:20.930482+0000","last_became_active":"2024-03-31T15:55:18.570424+0000","last_became_peered":"2024-03-31T15:55:18.570424+0000","last_unstale":"2024-03-31T15:55:20.930482+0000","last_undegraded":"2024-03-31T15:55:20.930482+0000","last_fullsized":"2024-03-31T15:55:20.930482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:13:15.572479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547692+0000","last_change":"2024-03-31T15:55:18.574827+0000","last_active":"2024-03-31T15:55:19.547692+0000","last_peered":"2024-03-31T15:55:19.547692+0000","last_clean":"2024-03-31T15:55:19.547692+0000","last_became_active":"2024-03-31T15:55:18.574539+0000","last_became_peered":"2024-03-31T15:55:18.574539+0000","last_unstale":"2024-03-31T15:55:19.547692+0000","last_undegraded":"2024-03-31T15:55:19.547692+0000","last_fullsized":"2024-03-31T15:55:19.547692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553659+0000","last_change":"2024-03-31T15:55:18.581864+0000","last_active":"2024-03-31T15:55:19.553659+0000","last_peered":"2024-03-31T15:55:19.553659+0000","last_clean":"2024-03-31T15:55:19.553659+0000","last_became_active":"2024-03-31T15:55:18.579446+0000","last_became_peered":"2024-03-31T15:55:18.579446+0000","last_unstale":"2024-03-31T15:55:19.553659+0000","last_undegraded":"2024-03-31T15:55:19.553659+0000","last_fullsized":"2024-03-31T15:55:19.553659+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:21.190346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547421+0000","last_change":"2024-03-31T15:55:18.581228+0000","last_active":"2024-03-31T15:55:19.547421+0000","last_peered":"2024-03-31T15:55:19.547421+0000","last_clean":"2024-03-31T15:55:19.547421+0000","last_became_active":"2024-03-31T15:55:18.580930+0000","last_became_peered":"2024-03-31T15:55:18.580930+0000","last_unstale":"2024-03-31T15:55:19.547421+0000","last_undegraded":"2024-03-31T15:55:19.547421+0000","last_fullsized":"2024-03-31T15:55:19.547421+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:48.538215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547145+0000","last_change":"2024-03-31T15:55:18.576569+0000","last_active":"2024-03-31T15:55:19.547145+0000","last_peered":"2024-03-31T15:55:19.547145+0000","last_clean":"2024-03-31T15:55:19.547145+0000","last_became_active":"2024-03-31T15:55:18.576267+0000","last_became_peered":"2024-03-31T15:55:18.576267+0000","last_unstale":"2024-03-31T15:55:19.547145+0000","last_undegraded":"2024-03-31T15:55:19.547145+0000","last_fullsized":"2024-03-31T15:55:19.547145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:02.259777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547964+0000","last_change":"2024-03-31T15:55:18.574506+0000","last_active":"2024-03-31T15:55:19.547964+0000","last_peered":"2024-03-31T15:55:19.547964+0000","last_clean":"2024-03-31T15:55:19.547964+0000","last_became_active":"2024-03-31T15:55:18.570090+0000","last_became_peered":"2024-03-31T15:55:18.570090+0000","last_unstale":"2024-03-31T15:55:19.547964+0000","last_undegraded":"2024-03-31T15:55:19.547964+0000","last_fullsized":"2024-03-31T15:55:19.547964+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:43:03.055246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930154+0000","last_change":"2024-03-31T15:55:18.560243+0000","last_active":"2024-03-31T15:55:20.930154+0000","last_peered":"2024-03-31T15:55:20.930154+0000","last_clean":"2024-03-31T15:55:20.930154+0000","last_became_active":"2024-03-31T15:55:18.559913+0000","last_became_peered":"2024-03-31T15:55:18.559913+0000","last_unstale":"2024-03-31T15:55:20.930154+0000","last_undegraded":"2024-03-31T15:55:20.930154+0000","last_fullsized":"2024-03-31T15:55:20.930154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548091+0000","last_change":"2024-03-31T15:55:18.570263+0000","last_active":"2024-03-31T15:55:19.548091+0000","last_peered":"2024-03-31T15:55:19.548091+0000","last_clean":"2024-03-31T15:55:19.548091+0000","last_became_active":"2024-03-31T15:55:18.569923+0000","last_became_peered":"2024-03-31T15:55:18.569923+0000","last_unstale":"2024-03-31T15:55:19.548091+0000","last_undegraded":"2024-03-31T15:55:19.548091+0000","last_fullsized":"2024-03-31T15:55:19.548091+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547746+0000","last_change":"2024-03-31T15:55:17.569534+0000","last_active":"2024-03-31T15:55:19.547746+0000","last_peered":"2024-03-31T15:55:19.547746+0000","last_clean":"2024-03-31T15:55:19.547746+0000","last_became_active":"2024-03-31T15:55:17.568983+0000","last_became_peered":"2024-03-31T15:55:17.568983+0000","last_unstale":"2024-03-31T15:55:19.547746+0000","last_undegraded":"2024-03-31T15:55:19.547746+0000","last_fullsized":"2024-03-31T15:55:19.547746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930469+0000","last_change":"2024-03-31T15:55:18.575181+0000","last_active":"2024-03-31T15:55:20.930469+0000","last_peered":"2024-03-31T15:55:20.930469+0000","last_clean":"2024-03-31T15:55:20.930469+0000","last_became_active":"2024-03-31T15:55:18.574878+0000","last_became_peered":"2024-03-31T15:55:18.574878+0000","last_unstale":"2024-03-31T15:55:20.930469+0000","last_undegraded":"2024-03-31T15:55:20.930469+0000","last_fullsized":"2024-03-31T15:55:20.930469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:29:27.178886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838089+0000","last_change":"2024-03-31T15:55:17.561836+0000","last_active":"2024-03-31T15:55:20.838089+0000","last_peered":"2024-03-31T15:55:20.838089+0000","last_clean":"2024-03-31T15:55:20.838089+0000","last_became_active":"2024-03-31T15:55:17.561561+0000","last_became_peered":"2024-03-31T15:55:17.561561+0000","last_unstale":"2024-03-31T15:55:20.838089+0000","last_undegraded":"2024-03-31T15:55:20.838089+0000","last_fullsized":"2024-03-31T15:55:20.838089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121107+0000","last_change":"2024-03-31T15:55:18.580605+0000","last_active":"2024-03-31T15:55:21.121107+0000","last_peered":"2024-03-31T15:55:21.121107+0000","last_clean":"2024-03-31T15:55:21.121107+0000","last_became_active":"2024-03-31T15:55:18.580334+0000","last_became_peered":"2024-03-31T15:55:18.580334+0000","last_unstale":"2024-03-31T15:55:21.121107+0000","last_undegraded":"2024-03-31T15:55:21.121107+0000","last_fullsized":"2024-03-31T15:55:21.121107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930625+0000","last_change":"2024-03-31T15:55:17.566422+0000","last_active":"2024-03-31T15:55:20.930625+0000","last_peered":"2024-03-31T15:55:20.930625+0000","last_clean":"2024-03-31T15:55:20.930625+0000","last_became_active":"2024-03-31T15:55:17.566154+0000","last_became_peered":"2024-03-31T15:55:17.566154+0000","last_unstale":"2024-03-31T15:55:20.930625+0000","last_undegraded":"2024-03-31T15:55:20.930625+0000","last_fullsized":"2024-03-31T15:55:20.930625+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:42:15.816385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548666+0000","last_change":"2024-03-31T15:55:18.583489+0000","last_active":"2024-03-31T15:55:19.548666+0000","last_peered":"2024-03-31T15:55:19.548666+0000","last_clean":"2024-03-31T15:55:19.548666+0000","last_became_active":"2024-03-31T15:55:18.583246+0000","last_became_peered":"2024-03-31T15:55:18.583246+0000","last_unstale":"2024-03-31T15:55:19.548666+0000","last_undegraded":"2024-03-31T15:55:19.548666+0000","last_fullsized":"2024-03-31T15:55:19.548666+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:48:46.491057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120947+0000","last_change":"2024-03-31T15:55:17.563636+0000","last_active":"2024-03-31T15:55:21.120947+0000","last_peered":"2024-03-31T15:55:21.120947+0000","last_clean":"2024-03-31T15:55:21.120947+0000","last_became_active":"2024-03-31T15:55:17.562976+0000","last_became_peered":"2024-03-31T15:55:17.562976+0000","last_unstale":"2024-03-31T15:55:21.120947+0000","last_undegraded":"2024-03-31T15:55:21.120947+0000","last_fullsized":"2024-03-31T15:55:21.120947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837772+0000","last_change":"2024-03-31T15:55:18.567428+0000","last_active":"2024-03-31T15:55:20.837772+0000","last_peered":"2024-03-31T15:55:20.837772+0000","last_clean":"2024-03-31T15:55:20.837772+0000","last_became_active":"2024-03-31T15:55:18.566876+0000","last_became_peered":"2024-03-31T15:55:18.566876+0000","last_unstale":"2024-03-31T15:55:20.837772+0000","last_undegraded":"2024-03-31T15:55:20.837772+0000","last_fullsized":"2024-03-31T15:55:20.837772+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553620+0000","last_change":"2024-03-31T15:55:17.564638+0000","last_active":"2024-03-31T15:55:19.553620+0000","last_peered":"2024-03-31T15:55:19.553620+0000","last_clean":"2024-03-31T15:55:19.553620+0000","last_became_active":"2024-03-31T15:55:17.564347+0000","last_became_peered":"2024-03-31T15:55:17.564347+0000","last_unstale":"2024-03-31T15:55:19.553620+0000","last_undegraded":"2024-03-31T15:55:19.553620+0000","last_fullsized":"2024-03-31T15:55:19.553620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:16:17.020690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553651+0000","last_change":"2024-03-31T15:55:18.567191+0000","last_active":"2024-03-31T15:55:19.553651+0000","last_peered":"2024-03-31T15:55:19.553651+0000","last_clean":"2024-03-31T15:55:19.553651+0000","last_became_active":"2024-03-31T15:55:18.566900+0000","last_became_peered":"2024-03-31T15:55:18.566900+0000","last_unstale":"2024-03-31T15:55:19.553651+0000","last_undegraded":"2024-03-31T15:55:19.553651+0000","last_fullsized":"2024-03-31T15:55:19.553651+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:29:03.221135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547845+0000","last_change":"2024-03-31T15:55:17.569775+0000","last_active":"2024-03-31T15:55:19.547845+0000","last_peered":"2024-03-31T15:55:19.547845+0000","last_clean":"2024-03-31T15:55:19.547845+0000","last_became_active":"2024-03-31T15:55:17.568127+0000","last_became_peered":"2024-03-31T15:55:17.568127+0000","last_unstale":"2024-03-31T15:55:19.547845+0000","last_undegraded":"2024-03-31T15:55:19.547845+0000","last_fullsized":"2024-03-31T15:55:19.547845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:19:15.971153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121038+0000","last_change":"2024-03-31T15:55:18.573156+0000","last_active":"2024-03-31T15:55:21.121038+0000","last_peered":"2024-03-31T15:55:21.121038+0000","last_clean":"2024-03-31T15:55:21.121038+0000","last_became_active":"2024-03-31T15:55:18.572316+0000","last_became_peered":"2024-03-31T15:55:18.572316+0000","last_unstale":"2024-03-31T15:55:21.121038+0000","last_undegraded":"2024-03-31T15:55:21.121038+0000","last_fullsized":"2024-03-31T15:55:21.121038+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:17.564811+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:17.564461+0000","last_became_peered":"2024-03-31T15:55:17.564461+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548413+0000","last_change":"2024-03-31T15:55:18.572911+0000","last_active":"2024-03-31T15:55:19.548413+0000","last_peered":"2024-03-31T15:55:19.548413+0000","last_clean":"2024-03-31T15:55:19.548413+0000","last_became_active":"2024-03-31T15:55:18.572663+0000","last_became_peered":"2024-03-31T15:55:18.572663+0000","last_unstale":"2024-03-31T15:55:19.548413+0000","last_undegraded":"2024-03-31T15:55:19.548413+0000","last_fullsized":"2024-03-31T15:55:19.548413+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120482+0000","last_change":"2024-03-31T15:55:17.558598+0000","last_active":"2024-03-31T15:55:21.120482+0000","last_peered":"2024-03-31T15:55:21.120482+0000","last_clean":"2024-03-31T15:55:21.120482+0000","last_became_active":"2024-03-31T15:55:17.557967+0000","last_became_peered":"2024-03-31T15:55:17.557967+0000","last_unstale":"2024-03-31T15:55:21.120482+0000","last_undegraded":"2024-03-31T15:55:21.120482+0000","last_fullsized":"2024-03-31T15:55:21.120482+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550725+0000","last_change":"2024-03-31T15:55:18.558312+0000","last_active":"2024-03-31T15:55:19.550725+0000","last_peered":"2024-03-31T15:55:19.550725+0000","last_clean":"2024-03-31T15:55:19.550725+0000","last_became_active":"2024-03-31T15:55:18.556133+0000","last_became_peered":"2024-03-31T15:55:18.556133+0000","last_unstale":"2024-03-31T15:55:19.550725+0000","last_undegraded":"2024-03-31T15:55:19.550725+0000","last_fullsized":"2024-03-31T15:55:19.550725+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:51:25.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548747+0000","last_change":"2024-03-31T15:55:17.559251+0000","last_active":"2024-03-31T15:55:19.548747+0000","last_peered":"2024-03-31T15:55:19.548747+0000","last_clean":"2024-03-31T15:55:19.548747+0000","last_became_active":"2024-03-31T15:55:17.558977+0000","last_became_peered":"2024-03-31T15:55:17.558977+0000","last_unstale":"2024-03-31T15:55:19.548747+0000","last_undegraded":"2024-03-31T15:55:19.548747+0000","last_fullsized":"2024-03-31T15:55:19.548747+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:17:23.098269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548185+0000","last_change":"2024-03-31T15:55:18.576163+0000","last_active":"2024-03-31T15:55:19.548185+0000","last_peered":"2024-03-31T15:55:19.548185+0000","last_clean":"2024-03-31T15:55:19.548185+0000","last_became_active":"2024-03-31T15:55:18.575491+0000","last_became_peered":"2024-03-31T15:55:18.575491+0000","last_unstale":"2024-03-31T15:55:19.548185+0000","last_undegraded":"2024-03-31T15:55:19.548185+0000","last_fullsized":"2024-03-31T15:55:19.548185+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:38:25.785072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553997+0000","last_change":"2024-03-31T15:55:17.559025+0000","last_active":"2024-03-31T15:55:19.553997+0000","last_peered":"2024-03-31T15:55:19.553997+0000","last_clean":"2024-03-31T15:55:19.553997+0000","last_became_active":"2024-03-31T15:55:17.558732+0000","last_became_peered":"2024-03-31T15:55:17.558732+0000","last_unstale":"2024-03-31T15:55:19.553997+0000","last_undegraded":"2024-03-31T15:55:19.553997+0000","last_fullsized":"2024-03-31T15:55:19.553997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:03.959563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838073+0000","last_change":"2024-03-31T15:55:18.559732+0000","last_active":"2024-03-31T15:55:20.838073+0000","last_peered":"2024-03-31T15:55:20.838073+0000","last_clean":"2024-03-31T15:55:20.838073+0000","last_became_active":"2024-03-31T15:55:18.559432+0000","last_became_peered":"2024-03-31T15:55:18.559432+0000","last_unstale":"2024-03-31T15:55:20.838073+0000","last_undegraded":"2024-03-31T15:55:20.838073+0000","last_fullsized":"2024-03-31T15:55:20.838073+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:15.200810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:17.559993+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:17.559669+0000","last_became_peered":"2024-03-31T15:55:17.559669+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:24:33.510910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121078+0000","last_change":"2024-03-31T15:55:18.558993+0000","last_active":"2024-03-31T15:55:21.121078+0000","last_peered":"2024-03-31T15:55:21.121078+0000","last_clean":"2024-03-31T15:55:21.121078+0000","last_became_active":"2024-03-31T15:55:18.558670+0000","last_became_peered":"2024-03-31T15:55:18.558670+0000","last_unstale":"2024-03-31T15:55:21.121078+0000","last_undegraded":"2024-03-31T15:55:21.121078+0000","last_fullsized":"2024-03-31T15:55:21.121078+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:54:50.792461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548239+0000","last_change":"2024-03-31T15:55:17.561703+0000","last_active":"2024-03-31T15:55:19.548239+0000","last_peered":"2024-03-31T15:55:19.548239+0000","last_clean":"2024-03-31T15:55:19.548239+0000","last_became_active":"2024-03-31T15:55:17.561448+0000","last_became_peered":"2024-03-31T15:55:17.561448+0000","last_unstale":"2024-03-31T15:55:19.548239+0000","last_undegraded":"2024-03-31T15:55:19.548239+0000","last_fullsized":"2024-03-31T15:55:19.548239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:41:38.684847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837750+0000","last_change":"2024-03-31T15:55:18.573704+0000","last_active":"2024-03-31T15:55:20.837750+0000","last_peered":"2024-03-31T15:55:20.837750+0000","last_clean":"2024-03-31T15:55:20.837750+0000","last_became_active":"2024-03-31T15:55:18.573462+0000","last_became_peered":"2024-03-31T15:55:18.573462+0000","last_unstale":"2024-03-31T15:55:20.837750+0000","last_undegraded":"2024-03-31T15:55:20.837750+0000","last_fullsized":"2024-03-31T15:55:20.837750+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:48:30.327007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548566+0000","last_change":"2024-03-31T15:55:17.565178+0000","last_active":"2024-03-31T15:55:19.548566+0000","last_peered":"2024-03-31T15:55:19.548566+0000","last_clean":"2024-03-31T15:55:19.548566+0000","last_became_active":"2024-03-31T15:55:17.564972+0000","last_became_peered":"2024-03-31T15:55:17.564972+0000","last_unstale":"2024-03-31T15:55:19.548566+0000","last_undegraded":"2024-03-31T15:55:19.548566+0000","last_fullsized":"2024-03-31T15:55:19.548566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:03:43.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548114+0000","last_change":"2024-03-31T15:55:18.576538+0000","last_active":"2024-03-31T15:55:19.548114+0000","last_peered":"2024-03-31T15:55:19.548114+0000","last_clean":"2024-03-31T15:55:19.548114+0000","last_became_active":"2024-03-31T15:55:18.569912+0000","last_became_peered":"2024-03-31T15:55:18.569912+0000","last_unstale":"2024-03-31T15:55:19.548114+0000","last_undegraded":"2024-03-31T15:55:19.548114+0000","last_fullsized":"2024-03-31T15:55:19.548114+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:42:18.227383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930236+0000","last_change":"2024-03-31T15:55:17.564498+0000","last_active":"2024-03-31T15:55:20.930236+0000","last_peered":"2024-03-31T15:55:20.930236+0000","last_clean":"2024-03-31T15:55:20.930236+0000","last_became_active":"2024-03-31T15:55:17.564148+0000","last_became_peered":"2024-03-31T15:55:17.564148+0000","last_unstale":"2024-03-31T15:55:20.930236+0000","last_undegraded":"2024-03-31T15:55:20.930236+0000","last_fullsized":"2024-03-31T15:55:20.930236+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:42:11.639287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547663+0000","last_change":"2024-03-31T15:55:18.567828+0000","last_active":"2024-03-31T15:55:19.547663+0000","last_peered":"2024-03-31T15:55:19.547663+0000","last_clean":"2024-03-31T15:55:19.547663+0000","last_became_active":"2024-03-31T15:55:18.567508+0000","last_became_peered":"2024-03-31T15:55:18.567508+0000","last_unstale":"2024-03-31T15:55:19.547663+0000","last_undegraded":"2024-03-31T15:55:19.547663+0000","last_fullsized":"2024-03-31T15:55:19.547663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:09.454583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547389+0000","last_change":"2024-03-31T15:55:17.568236+0000","last_active":"2024-03-31T15:55:19.547389+0000","last_peered":"2024-03-31T15:55:19.547389+0000","last_clean":"2024-03-31T15:55:19.547389+0000","last_became_active":"2024-03-31T15:55:17.567952+0000","last_became_peered":"2024-03-31T15:55:17.567952+0000","last_unstale":"2024-03-31T15:55:19.547389+0000","last_undegraded":"2024-03-31T15:55:19.547389+0000","last_fullsized":"2024-03-31T15:55:19.547389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:49:10.892805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548449+0000","last_change":"2024-03-31T15:55:18.570563+0000","last_active":"2024-03-31T15:55:19.548449+0000","last_peered":"2024-03-31T15:55:19.548449+0000","last_clean":"2024-03-31T15:55:19.548449+0000","last_became_active":"2024-03-31T15:55:18.570292+0000","last_became_peered":"2024-03-31T15:55:18.570292+0000","last_unstale":"2024-03-31T15:55:19.548449+0000","last_undegraded":"2024-03-31T15:55:19.548449+0000","last_fullsized":"2024-03-31T15:55:19.548449+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:33:59.095581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547181+0000","last_change":"2024-03-31T15:55:17.568544+0000","last_active":"2024-03-31T15:55:19.547181+0000","last_peered":"2024-03-31T15:55:19.547181+0000","last_clean":"2024-03-31T15:55:19.547181+0000","last_became_active":"2024-03-31T15:55:17.567942+0000","last_became_peered":"2024-03-31T15:55:17.567942+0000","last_unstale":"2024-03-31T15:55:19.547181+0000","last_undegraded":"2024-03-31T15:55:19.547181+0000","last_fullsized":"2024-03-31T15:55:19.547181+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:10.013322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548171+0000","last_change":"2024-03-31T15:55:18.574899+0000","last_active":"2024-03-31T15:55:19.548171+0000","last_peered":"2024-03-31T15:55:19.548171+0000","last_clean":"2024-03-31T15:55:19.548171+0000","last_became_active":"2024-03-31T15:55:18.574347+0000","last_became_peered":"2024-03-31T15:55:18.574347+0000","last_unstale":"2024-03-31T15:55:19.548171+0000","last_undegraded":"2024-03-31T15:55:19.548171+0000","last_fullsized":"2024-03-31T15:55:19.548171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554076+0000","last_change":"2024-03-31T15:55:17.564758+0000","last_active":"2024-03-31T15:55:19.554076+0000","last_peered":"2024-03-31T15:55:19.554076+0000","last_clean":"2024-03-31T15:55:19.554076+0000","last_became_active":"2024-03-31T15:55:17.564417+0000","last_became_peered":"2024-03-31T15:55:17.564417+0000","last_unstale":"2024-03-31T15:55:19.554076+0000","last_undegraded":"2024-03-31T15:55:19.554076+0000","last_fullsized":"2024-03-31T15:55:19.554076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:01.251935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553903+0000","last_change":"2024-03-31T15:55:18.569677+0000","last_active":"2024-03-31T15:55:19.553903+0000","last_peered":"2024-03-31T15:55:19.553903+0000","last_clean":"2024-03-31T15:55:19.553903+0000","last_became_active":"2024-03-31T15:55:18.569325+0000","last_became_peered":"2024-03-31T15:55:18.569325+0000","last_unstale":"2024-03-31T15:55:19.553903+0000","last_undegraded":"2024-03-31T15:55:19.553903+0000","last_fullsized":"2024-03-31T15:55:19.553903+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:50:35.297846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.121071+0000","last_change":"2024-03-31T15:55:17.563289+0000","last_active":"2024-03-31T15:55:21.121071+0000","last_peered":"2024-03-31T15:55:21.121071+0000","last_clean":"2024-03-31T15:55:21.121071+0000","last_became_active":"2024-03-31T15:55:17.562945+0000","last_became_peered":"2024-03-31T15:55:17.562945+0000","last_unstale":"2024-03-31T15:55:21.121071+0000","last_undegraded":"2024-03-31T15:55:21.121071+0000","last_fullsized":"2024-03-31T15:55:21.121071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120928+0000","last_change":"2024-03-31T15:55:18.557599+0000","last_active":"2024-03-31T15:55:21.120928+0000","last_peered":"2024-03-31T15:55:21.120928+0000","last_clean":"2024-03-31T15:55:21.120928+0000","last_became_active":"2024-03-31T15:55:18.557275+0000","last_became_peered":"2024-03-31T15:55:18.557275+0000","last_unstale":"2024-03-31T15:55:21.120928+0000","last_undegraded":"2024-03-31T15:55:21.120928+0000","last_fullsized":"2024-03-31T15:55:21.120928+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:22:15.955038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550772+0000","last_change":"2024-03-31T15:55:17.562464+0000","last_active":"2024-03-31T15:55:19.550772+0000","last_peered":"2024-03-31T15:55:19.550772+0000","last_clean":"2024-03-31T15:55:19.550772+0000","last_became_active":"2024-03-31T15:55:17.562192+0000","last_became_peered":"2024-03-31T15:55:17.562192+0000","last_unstale":"2024-03-31T15:55:19.550772+0000","last_undegraded":"2024-03-31T15:55:19.550772+0000","last_fullsized":"2024-03-31T15:55:19.550772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:13.323516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550931+0000","last_change":"2024-03-31T15:55:18.572667+0000","last_active":"2024-03-31T15:55:19.550931+0000","last_peered":"2024-03-31T15:55:19.550931+0000","last_clean":"2024-03-31T15:55:19.550931+0000","last_became_active":"2024-03-31T15:55:18.571226+0000","last_became_peered":"2024-03-31T15:55:18.571226+0000","last_unstale":"2024-03-31T15:55:19.550931+0000","last_undegraded":"2024-03-31T15:55:19.550931+0000","last_fullsized":"2024-03-31T15:55:19.550931+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:54:13.587576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930339+0000","last_change":"2024-03-31T15:55:17.564482+0000","last_active":"2024-03-31T15:55:20.930339+0000","last_peered":"2024-03-31T15:55:20.930339+0000","last_clean":"2024-03-31T15:55:20.930339+0000","last_became_active":"2024-03-31T15:55:17.564263+0000","last_became_peered":"2024-03-31T15:55:17.564263+0000","last_unstale":"2024-03-31T15:55:20.930339+0000","last_undegraded":"2024-03-31T15:55:20.930339+0000","last_fullsized":"2024-03-31T15:55:20.930339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:09:21.691754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548137+0000","last_change":"2024-03-31T15:55:18.573739+0000","last_active":"2024-03-31T15:55:19.548137+0000","last_peered":"2024-03-31T15:55:19.548137+0000","last_clean":"2024-03-31T15:55:19.548137+0000","last_became_active":"2024-03-31T15:55:18.573288+0000","last_became_peered":"2024-03-31T15:55:18.573288+0000","last_unstale":"2024-03-31T15:55:19.548137+0000","last_undegraded":"2024-03-31T15:55:19.548137+0000","last_fullsized":"2024-03-31T15:55:19.548137+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:00:18.548155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550590+0000","last_change":"2024-03-31T15:55:17.561760+0000","last_active":"2024-03-31T15:55:19.550590+0000","last_peered":"2024-03-31T15:55:19.550590+0000","last_clean":"2024-03-31T15:55:19.550590+0000","last_became_active":"2024-03-31T15:55:17.561471+0000","last_became_peered":"2024-03-31T15:55:17.561471+0000","last_unstale":"2024-03-31T15:55:19.550590+0000","last_undegraded":"2024-03-31T15:55:19.550590+0000","last_fullsized":"2024-03-31T15:55:19.550590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:31:27.557663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837752+0000","last_change":"2024-03-31T15:55:18.557363+0000","last_active":"2024-03-31T15:55:20.837752+0000","last_peered":"2024-03-31T15:55:20.837752+0000","last_clean":"2024-03-31T15:55:20.837752+0000","last_became_active":"2024-03-31T15:55:18.557133+0000","last_became_peered":"2024-03-31T15:55:18.557133+0000","last_unstale":"2024-03-31T15:55:20.837752+0000","last_undegraded":"2024-03-31T15:55:20.837752+0000","last_fullsized":"2024-03-31T15:55:20.837752+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550455+0000","last_change":"2024-03-31T15:55:17.567522+0000","last_active":"2024-03-31T15:55:19.550455+0000","last_peered":"2024-03-31T15:55:19.550455+0000","last_clean":"2024-03-31T15:55:19.550455+0000","last_became_active":"2024-03-31T15:55:17.567296+0000","last_became_peered":"2024-03-31T15:55:17.567296+0000","last_unstale":"2024-03-31T15:55:19.550455+0000","last_undegraded":"2024-03-31T15:55:19.550455+0000","last_fullsized":"2024-03-31T15:55:19.550455+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:24:18.305487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553568+0000","last_change":"2024-03-31T15:55:18.578437+0000","last_active":"2024-03-31T15:55:19.553568+0000","last_peered":"2024-03-31T15:55:19.553568+0000","last_clean":"2024-03-31T15:55:19.553568+0000","last_became_active":"2024-03-31T15:55:18.578122+0000","last_became_peered":"2024-03-31T15:55:18.578122+0000","last_unstale":"2024-03-31T15:55:19.553568+0000","last_undegraded":"2024-03-31T15:55:19.553568+0000","last_fullsized":"2024-03-31T15:55:19.553568+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:07:24.502072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548305+0000","last_change":"2024-03-31T15:55:17.569573+0000","last_active":"2024-03-31T15:55:19.548305+0000","last_peered":"2024-03-31T15:55:19.548305+0000","last_clean":"2024-03-31T15:55:19.548305+0000","last_became_active":"2024-03-31T15:55:17.569295+0000","last_became_peered":"2024-03-31T15:55:17.569295+0000","last_unstale":"2024-03-31T15:55:19.548305+0000","last_undegraded":"2024-03-31T15:55:19.548305+0000","last_fullsized":"2024-03-31T15:55:19.548305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:23:11.923571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548574+0000","last_change":"2024-03-31T15:55:18.581151+0000","last_active":"2024-03-31T15:55:19.548574+0000","last_peered":"2024-03-31T15:55:19.548574+0000","last_clean":"2024-03-31T15:55:19.548574+0000","last_became_active":"2024-03-31T15:55:18.580843+0000","last_became_peered":"2024-03-31T15:55:18.580843+0000","last_unstale":"2024-03-31T15:55:19.548574+0000","last_undegraded":"2024-03-31T15:55:19.548574+0000","last_fullsized":"2024-03-31T15:55:19.548574+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.550481+0000","last_change":"2024-03-31T15:55:17.561569+0000","last_active":"2024-03-31T15:55:19.550481+0000","last_peered":"2024-03-31T15:55:19.550481+0000","last_clean":"2024-03-31T15:55:19.550481+0000","last_became_active":"2024-03-31T15:55:17.560629+0000","last_became_peered":"2024-03-31T15:55:17.560629+0000","last_unstale":"2024-03-31T15:55:19.550481+0000","last_undegraded":"2024-03-31T15:55:19.550481+0000","last_fullsized":"2024-03-31T15:55:19.550481+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:36.826694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930424+0000","last_change":"2024-03-31T15:55:18.569951+0000","last_active":"2024-03-31T15:55:20.930424+0000","last_peered":"2024-03-31T15:55:20.930424+0000","last_clean":"2024-03-31T15:55:20.930424+0000","last_became_active":"2024-03-31T15:55:18.569622+0000","last_became_peered":"2024-03-31T15:55:18.569622+0000","last_unstale":"2024-03-31T15:55:20.930424+0000","last_undegraded":"2024-03-31T15:55:20.930424+0000","last_fullsized":"2024-03-31T15:55:20.930424+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.554129+0000","last_change":"2024-03-31T15:55:17.558967+0000","last_active":"2024-03-31T15:55:19.554129+0000","last_peered":"2024-03-31T15:55:19.554129+0000","last_clean":"2024-03-31T15:55:19.554129+0000","last_became_active":"2024-03-31T15:55:17.558642+0000","last_became_peered":"2024-03-31T15:55:17.558642+0000","last_unstale":"2024-03-31T15:55:19.554129+0000","last_undegraded":"2024-03-31T15:55:19.554129+0000","last_fullsized":"2024-03-31T15:55:19.554129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:09:47.530373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548388+0000","last_change":"2024-03-31T15:55:18.576406+0000","last_active":"2024-03-31T15:55:19.548388+0000","last_peered":"2024-03-31T15:55:19.548388+0000","last_clean":"2024-03-31T15:55:19.548388+0000","last_became_active":"2024-03-31T15:55:18.575892+0000","last_became_peered":"2024-03-31T15:55:18.575892+0000","last_unstale":"2024-03-31T15:55:19.548388+0000","last_undegraded":"2024-03-31T15:55:19.548388+0000","last_fullsized":"2024-03-31T15:55:19.548388+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:07:56.723595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.552589+0000","last_change":"2024-03-31T15:55:17.565054+0000","last_active":"2024-03-31T15:55:19.552589+0000","last_peered":"2024-03-31T15:55:19.552589+0000","last_clean":"2024-03-31T15:55:19.552589+0000","last_became_active":"2024-03-31T15:55:17.564845+0000","last_became_peered":"2024-03-31T15:55:17.564845+0000","last_unstale":"2024-03-31T15:55:19.552589+0000","last_undegraded":"2024-03-31T15:55:19.552589+0000","last_fullsized":"2024-03-31T15:55:19.552589+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:34:03.576042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548610+0000","last_change":"2024-03-31T15:55:18.572781+0000","last_active":"2024-03-31T15:55:19.548610+0000","last_peered":"2024-03-31T15:55:19.548610+0000","last_clean":"2024-03-31T15:55:19.548610+0000","last_became_active":"2024-03-31T15:55:18.572475+0000","last_became_peered":"2024-03-31T15:55:18.572475+0000","last_unstale":"2024-03-31T15:55:19.548610+0000","last_undegraded":"2024-03-31T15:55:19.548610+0000","last_fullsized":"2024-03-31T15:55:19.548610+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:21.840147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.838034+0000","last_change":"2024-03-31T15:55:17.552434+0000","last_active":"2024-03-31T15:55:20.838034+0000","last_peered":"2024-03-31T15:55:20.838034+0000","last_clean":"2024-03-31T15:55:20.838034+0000","last_became_active":"2024-03-31T15:55:17.552137+0000","last_became_peered":"2024-03-31T15:55:17.552137+0000","last_unstale":"2024-03-31T15:55:20.838034+0000","last_undegraded":"2024-03-31T15:55:20.838034+0000","last_fullsized":"2024-03-31T15:55:20.838034+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:59:00.296763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837915+0000","last_change":"2024-03-31T15:55:18.567217+0000","last_active":"2024-03-31T15:55:20.837915+0000","last_peered":"2024-03-31T15:55:20.837915+0000","last_clean":"2024-03-31T15:55:20.837915+0000","last_became_active":"2024-03-31T15:55:18.566527+0000","last_became_peered":"2024-03-31T15:55:18.566527+0000","last_unstale":"2024-03-31T15:55:20.837915+0000","last_undegraded":"2024-03-31T15:55:20.837915+0000","last_fullsized":"2024-03-31T15:55:20.837915+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:51:35.051000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.837837+0000","last_change":"2024-03-31T15:55:17.569970+0000","last_active":"2024-03-31T15:55:20.837837+0000","last_peered":"2024-03-31T15:55:20.837837+0000","last_clean":"2024-03-31T15:55:20.837837+0000","last_became_active":"2024-03-31T15:55:17.569178+0000","last_became_peered":"2024-03-31T15:55:17.569178+0000","last_unstale":"2024-03-31T15:55:20.837837+0000","last_undegraded":"2024-03-31T15:55:20.837837+0000","last_fullsized":"2024-03-31T15:55:20.837837+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:20.930107+0000","last_change":"2024-03-31T15:55:18.576213+0000","last_active":"2024-03-31T15:55:20.930107+0000","last_peered":"2024-03-31T15:55:20.930107+0000","last_clean":"2024-03-31T15:55:20.930107+0000","last_became_active":"2024-03-31T15:55:18.575930+0000","last_became_peered":"2024-03-31T15:55:18.575930+0000","last_unstale":"2024-03-31T15:55:20.930107+0000","last_undegraded":"2024-03-31T15:55:20.930107+0000","last_fullsized":"2024-03-31T15:55:20.930107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:33:20.480648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547246+0000","last_change":"2024-03-31T15:55:17.569351+0000","last_active":"2024-03-31T15:55:19.547246+0000","last_peered":"2024-03-31T15:55:19.547246+0000","last_clean":"2024-03-31T15:55:19.547246+0000","last_became_active":"2024-03-31T15:55:17.569079+0000","last_became_peered":"2024-03-31T15:55:17.569079+0000","last_unstale":"2024-03-31T15:55:19.547246+0000","last_undegraded":"2024-03-31T15:55:19.547246+0000","last_fullsized":"2024-03-31T15:55:19.547246+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:23:34.483640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120705+0000","last_change":"2024-03-31T15:55:18.572994+0000","last_active":"2024-03-31T15:55:21.120705+0000","last_peered":"2024-03-31T15:55:21.120705+0000","last_clean":"2024-03-31T15:55:21.120705+0000","last_became_active":"2024-03-31T15:55:18.572692+0000","last_became_peered":"2024-03-31T15:55:18.572692+0000","last_unstale":"2024-03-31T15:55:21.120705+0000","last_undegraded":"2024-03-31T15:55:21.120705+0000","last_fullsized":"2024-03-31T15:55:21.120705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:17:04.967404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:21.120545+0000","last_change":"2024-03-31T15:55:17.563312+0000","last_active":"2024-03-31T15:55:21.120545+0000","last_peered":"2024-03-31T15:55:21.120545+0000","last_clean":"2024-03-31T15:55:21.120545+0000","last_became_active":"2024-03-31T15:55:17.562982+0000","last_became_peered":"2024-03-31T15:55:17.562982+0000","last_unstale":"2024-03-31T15:55:21.120545+0000","last_undegraded":"2024-03-31T15:55:21.120545+0000","last_fullsized":"2024-03-31T15:55:21.120545+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:51:54.669312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548081+0000","last_change":"2024-03-31T15:55:18.575859+0000","last_active":"2024-03-31T15:55:19.548081+0000","last_peered":"2024-03-31T15:55:19.548081+0000","last_clean":"2024-03-31T15:55:19.548081+0000","last_became_active":"2024-03-31T15:55:18.575507+0000","last_became_peered":"2024-03-31T15:55:18.575507+0000","last_unstale":"2024-03-31T15:55:19.548081+0000","last_undegraded":"2024-03-31T15:55:19.548081+0000","last_fullsized":"2024-03-31T15:55:19.548081+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:26:59.904145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.547913+0000","last_change":"2024-03-31T15:55:17.559862+0000","last_active":"2024-03-31T15:55:19.547913+0000","last_peered":"2024-03-31T15:55:19.547913+0000","last_clean":"2024-03-31T15:55:19.547913+0000","last_became_active":"2024-03-31T15:55:17.558616+0000","last_became_peered":"2024-03-31T15:55:17.558616+0000","last_unstale":"2024-03-31T15:55:19.547913+0000","last_undegraded":"2024-03-31T15:55:19.547913+0000","last_fullsized":"2024-03-31T15:55:19.547913+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:45:28.724128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.553996+0000","last_change":"2024-03-31T15:55:18.582994+0000","last_active":"2024-03-31T15:55:19.553996+0000","last_peered":"2024-03-31T15:55:19.553996+0000","last_clean":"2024-03-31T15:55:19.553996+0000","last_became_active":"2024-03-31T15:55:18.580575+0000","last_became_peered":"2024-03-31T15:55:18.580575+0000","last_unstale":"2024-03-31T15:55:19.553996+0000","last_undegraded":"2024-03-31T15:55:19.553996+0000","last_fullsized":"2024-03-31T15:55:19.553996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:17.532502+0000","last_clean_scrub_stamp":"2024-03-31T15:55:17.532502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:02:59.330574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548059+0000","last_change":"2024-03-31T15:55:17.570487+0000","last_active":"2024-03-31T15:55:19.548059+0000","last_peered":"2024-03-31T15:55:19.548059+0000","last_clean":"2024-03-31T15:55:19.548059+0000","last_became_active":"2024-03-31T15:55:17.570204+0000","last_became_peered":"2024-03-31T15:55:17.570204+0000","last_unstale":"2024-03-31T15:55:19.548059+0000","last_undegraded":"2024-03-31T15:55:19.548059+0000","last_fullsized":"2024-03-31T15:55:19.548059+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:55:16.524660+0000","last_clean_scrub_stamp":"2024-03-31T15:55:16.524660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:16:34.516949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-31T15:55:19.548316+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:55:19.548316+0000","last_peered":"2024-03-31T15:55:19.548316+0000","last_clean":"2024-03-31T15:55:19.548316+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:55:19.548316+0000","last_undegraded":"2024-03-31T15:55:19.548316+0000","last_fullsized":"2024-03-31T15:55:19.548316+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509465,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":446464,"data_stored":192513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509465,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":450560,"data_stored":192043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":192535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509466,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":192021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509466,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1040384,"data_stored":782333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542169,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":193577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542169,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":192001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1052672,"data_stored":783981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:55:27.644 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:55:27.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:55:27.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 Processor -- start 2024-03-31T15:55:27.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 -- start start 2024-03-31T15:55:27.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f0806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3f080cdf20 0x7f3f0806d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0815e8c0 con 0x7f3f0806ce50 2024-03-31T15:55:27.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f3f080d0ff0 con 0x7f3f080cdf20 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0f600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3f080d12d0 con 0x7f3f0815e4e0 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3f080cdf20 0x7f3f0806d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f3f080cdf20 0x7f3f0806d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50918/0 (socket says 172.21.15.90:50918) 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0cc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f0806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 -- 172.21.15.90:0/2142296476 learned_addr learned my addr 172.21.15.90:0/2142296476 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:27.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 -- 172.21.15.90:0/2142296476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 msgr2=0x7f3f0806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:27.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 --2- 172.21.15.90:0/2142296476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f0806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 -- 172.21.15.90:0/2142296476 >> v1:172.21.15.102:6789/0 conn(0x7f3f0806ce50 legacy=0x7f3f080ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.832+0000 7f3f0d600640 1 -- 172.21.15.90:0/2142296476 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f080cc710 con 0x7f3f080cdf20 2024-03-31T15:55:27.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.836+0000 7f3f0d600640 1 --2- 172.21.15.90:0/2142296476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f0806d290 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009e50 tx=0x7f3f0001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e6c43f1924b00a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.836+0000 7f3f07400640 1 -- 172.21.15.90:0/2142296476 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000a5660 con 0x7f3f080cdf20 2024-03-31T15:55:27.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.836+0000 7f3f07400640 1 -- 172.21.15.90:0/2142296476 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f000a5800 con 0x7f3f080cdf20 2024-03-31T15:55:27.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.836+0000 7f3f07400640 1 -- 172.21.15.90:0/2142296476 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f000a5bd0 con 0x7f3f080cdf20 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- 172.21.15.90:0/2142296476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 msgr2=0x7f3f0806d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- 172.21.15.90:0/2142296476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f0806d290 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3f00009e50 tx=0x7f3f0001c9a0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- 172.21.15.90:0/2142296476 shutdown_connections 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- 172.21.15.90:0/2142296476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f0806d290 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- 172.21.15.90:0/2142296476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f0806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- 172.21.15.90:0/2142296476 >> 172.21.15.90:0/2142296476 conn(0x7f3f080d1f20 msgr2=0x7f3f080d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:27.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- 172.21.15.90:0/2142296476 shutdown_connections 2024-03-31T15:55:27.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- 172.21.15.90:0/2142296476 wait complete. 2024-03-31T15:55:27.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 Processor -- start 2024-03-31T15:55:27.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- start start 2024-03-31T15:55:27.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0806ce50 0x7f3f08155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f080c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0816b1b0 con 0x7f3f080cdf20 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0816b030 con 0x7f3f0806ce50 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0f600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0816b330 con 0x7f3f0815e4e0 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f07e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50924/0 (socket says 172.21.15.90:50924) 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f07e00640 1 -- 172.21.15.90:0/3117356702 learned_addr learned my addr 172.21.15.90:0/3117356702 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0d600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f080c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.840+0000 7f3f0cc00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0806ce50 0x7f3f08155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 -- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 msgr2=0x7f3f080c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f080c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 -- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0806ce50 msgr2=0x7f3f08155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0806ce50 0x7f3f08155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:27.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 -- 172.21.15.90:0/3117356702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f080cbc70 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f07e00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c370 tx=0x7f3ef400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40052d0 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef4005470 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4007260 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f080cbf50 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f08155080 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f080685f0 con 0x7f3f080cdf20 2024-03-31T15:55:27.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.844+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3ef4007400 con 0x7f3f080cdf20 2024-03-31T15:55:27.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.848+0000 7f3f04c00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 0x7f3ee806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:27.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.848+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f3ef4085e70 con 0x7f3f080cdf20 2024-03-31T15:55:27.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.848+0000 7f3f0cc00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 0x7f3ee806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:27.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.848+0000 7f3f0cc00640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 0x7f3ee806a870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009b60 tx=0x7f3efc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:27.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:27.848+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3ef4055ce0 con 0x7f3f080cdf20 2024-03-31T15:55:28.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.212+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f3f0815afa0 con 0x7f3f080cdf20 2024-03-31T15:55:28.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.216+0000 7f3f04c00640 1 -- 172.21.15.90:0/3117356702 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f3ef4059fe0 con 0x7f3f080cdf20 2024-03-31T15:55:28.210 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:55:28.210 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:55:28.210 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:55:28.211 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:55:28.211 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:55:28.211 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:55:28.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 msgr2=0x7f3ee806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 0x7f3ee806a870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009b60 tx=0x7f3efc009190 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 msgr2=0x7f3f080c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3ef400c370 tx=0x7f3ef400c830 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 shutdown_connections 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ee80683b0 0x7f3ee806a870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f0815e4e0 0x7f3f080c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3f080cdf20 0x7f3f080c7b20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 --2- 172.21.15.90:0/3117356702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0806ce50 0x7f3f08155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 >> 172.21.15.90:0/3117356702 conn(0x7f3f080d1f20 msgr2=0x7f3f0806d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 shutdown_connections 2024-03-31T15:55:28.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.220+0000 7f3f0f600640 1 -- 172.21.15.90:0/3117356702 wait complete. 2024-03-31T15:55:28.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.372+0000 7f9ad6e00640 1 Processor -- start 2024-03-31T15:55:28.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad6e00640 1 -- start start 2024-03-31T15:55:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad6e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9ad006d7c0 0x7f9ad015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad6e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad015f780 con 0x7f9ad015e1b0 2024-03-31T15:55:28.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad6e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad015f1c0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad6e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad01622b0 con 0x7f9ad006d7c0 2024-03-31T15:55:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9acfe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9ad015e1b0 0x7f9ad0162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44310/0 (socket says 172.21.15.90:44310) 2024-03-31T15:55:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9acfe00640 1 -- 172.21.15.90:0/3402227069 learned_addr learned my addr 172.21.15.90:0/3402227069 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad4e00640 1 --2- 172.21.15.90:0/3402227069 >> v2:172.21.15.90:3300/0 conn(0x7f9ad006d7c0 0x7f9ad015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:28.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9acea00640 1 -- 172.21.15.90:0/3402227069 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164977804 0 0) 0x7f9ad015f780 con 0x7f9ad015e1b0 2024-03-31T15:55:28.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9acea00640 1 -- 172.21.15.90:0/3402227069 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab4003530 con 0x7f9ad015e1b0 2024-03-31T15:55:28.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad4e00640 1 -- 172.21.15.90:0/3402227069 >> v1:172.21.15.102:6789/0 conn(0x7f9ad006c7b0 legacy=0x7f9ad006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad4e00640 1 -- 172.21.15.90:0/3402227069 >> v1:172.21.15.90:6789/0 conn(0x7f9ad015e1b0 legacy=0x7f9ad0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.376+0000 7f9ad4e00640 1 -- 172.21.15.90:0/3402227069 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad015f4a0 con 0x7f9ad006d7c0 2024-03-31T15:55:28.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad4e00640 1 --2- 172.21.15.90:0/3402227069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006d7c0 0x7f9ad015e590 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009d90 tx=0x7f9ac801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272740b06773affb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:28.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9acea00640 1 -- 172.21.15.90:0/3402227069 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80b2070 con 0x7f9ad006d7c0 2024-03-31T15:55:28.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9acea00640 1 -- 172.21.15.90:0/3402227069 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac80a5970 con 0x7f9ad006d7c0 2024-03-31T15:55:28.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9acea00640 1 -- 172.21.15.90:0/3402227069 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80a5d40 con 0x7f9ad006d7c0 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- 172.21.15.90:0/3402227069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006d7c0 msgr2=0x7f9ad015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/3402227069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006d7c0 0x7f9ad015e590 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009d90 tx=0x7f9ac801c8b0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- 172.21.15.90:0/3402227069 shutdown_connections 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/3402227069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006d7c0 0x7f9ad015e590 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- 172.21.15.90:0/3402227069 >> 172.21.15.90:0/3402227069 conn(0x7f9ad009aa40 msgr2=0x7f9ad009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- 172.21.15.90:0/3402227069 shutdown_connections 2024-03-31T15:55:28.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- 172.21.15.90:0/3402227069 wait complete. 2024-03-31T15:55:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 Processor -- start 2024-03-31T15:55:28.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.380+0000 7f9ad6e00640 1 -- start start 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ad006d7c0 0x7f9ad00b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 0x7f9ad00cc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad00ce7a0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad00d2a50 con 0x7f9ad015e1b0 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad00ce920 con 0x7f9ad006d7c0 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50932/0 (socket says 172.21.15.90:50932) 2024-03-31T15:55:28.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 -- 172.21.15.90:0/1304737982 learned_addr learned my addr 172.21.15.90:0/1304737982 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad4e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 0x7f9ad00cc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acf400640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ad006d7c0 0x7f9ad00b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 -- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ad006d7c0 msgr2=0x7f9ad00b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ad006d7c0 0x7f9ad00b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 -- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 msgr2=0x7f9ad00cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 0x7f9ad00cc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 -- 172.21.15.90:0/1304737982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad00cca90 con 0x7f9ad006c7b0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad4e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 0x7f9ad00cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9acfe00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000c450 tx=0x7f9ac000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0017070 con 0x7f9ad006c7b0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac0005cf0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad00ccd10 con 0x7f9ad006c7b0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0007270 con 0x7f9ad006c7b0 2024-03-31T15:55:28.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad00cd2d0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad00686d0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.384+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ac001a070 con 0x7f9ad006c7b0 2024-03-31T15:55:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.388+0000 7f9ac7e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 0x7f9ab006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.388+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9ac0085db0 con 0x7f9ad006c7b0 2024-03-31T15:55:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.388+0000 7f9acf400640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 0x7f9ab006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:28.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.388+0000 7f9acf400640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 0x7f9ab006a840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ad015d010 tx=0x7f9abc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:28.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.392+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9ac008a070 con 0x7f9ad006c7b0 2024-03-31T15:55:28.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.580+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ad00b1b60 con 0x7f9ad006c7b0 2024-03-31T15:55:28.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.580+0000 7f9ac7e00640 1 -- 172.21.15.90:0/1304737982 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5184 (secure 0 0 0) 0x7f9ad00cca90 con 0x7f9ad006c7b0 2024-03-31T15:55:28.576 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:28.576 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:20.953376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:55:28.577 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 17 2024-03-31T15:55:28.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 msgr2=0x7f9ab006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 0x7f9ab006a840 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ad015d010 tx=0x7f9abc009040 comp rx=0 tx=0).stop 2024-03-31T15:55:28.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 msgr2=0x7f9ad014ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:28.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9ac000c450 tx=0x7f9ac000c910 comp rx=0 tx=0).stop 2024-03-31T15:55:28.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 shutdown_connections 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9ab0068380 0x7f9ab006a840 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ad015e1b0 0x7f9ad00cc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9ad006d7c0 0x7f9ad00b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 --2- 172.21.15.90:0/1304737982 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ad006c7b0 0x7f9ad014ff40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 >> 172.21.15.90:0/1304737982 conn(0x7f9ad009aa40 msgr2=0x7f9ad006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 shutdown_connections 2024-03-31T15:55:28.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:28.584+0000 7f9ad6e00640 1 -- 172.21.15.90:0/1304737982 wait complete. 2024-03-31T15:55:29.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:55:17.930644+0000', 'modified': '2024-03-31T15:55:20.953376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5502}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5502': {'gid': 5502, 'name': 'c', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6834/353199836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6832', 'nonce': 353199836}, {'type': 'v1', 'addr': '172.21.15.90:6834', 'nonce': 353199836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5502, 'qdb_cluster': [5502]} 2024-03-31T15:55:29.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:55:29.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-31T15:55:29.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.144+0000 7f9840200640 1 Processor -- start 2024-03-31T15:55:29.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.144+0000 7f9840200640 1 -- start start 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f9840200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98381624a0 0x7f98380d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f9840200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f98380d2940 0x7f98380b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f9840200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98380d1400 con 0x7f983815e320 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f9840200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f983815e700 con 0x7f98381624a0 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f9840200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f98380b1a20 con 0x7f98380d2940 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98381624a0 0x7f98380d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98381624a0 0x7f98380d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50946/0 (socket says 172.21.15.90:50946) 2024-03-31T15:55:29.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 -- 172.21.15.90:0/2560293656 learned_addr learned my addr 172.21.15.90:0/2560293656 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 -- 172.21.15.90:0/2560293656 >> v2:172.21.15.102:3300/0 conn(0x7f98380d2940 msgr2=0x7f98380b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 --2- 172.21.15.90:0/2560293656 >> v2:172.21.15.102:3300/0 conn(0x7f98380d2940 0x7f98380b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 -- 172.21.15.90:0/2560293656 >> v1:172.21.15.102:6789/0 conn(0x7f983815e320 legacy=0x7f9838161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.148+0000 7f983ce00640 1 -- 172.21.15.90:0/2560293656 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98380b1d00 con 0x7f98381624a0 2024-03-31T15:55:29.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f983ce00640 1 --2- 172.21.15.90:0/2560293656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98381624a0 0x7f98380d0d20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f98380b7fb0 tx=0x7f982401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2bd67fb1725249 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9837e00640 1 -- 172.21.15.90:0/2560293656 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240aa660 con 0x7f98381624a0 2024-03-31T15:55:29.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9837e00640 1 -- 172.21.15.90:0/2560293656 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98240af070 con 0x7f98381624a0 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9837e00640 1 -- 172.21.15.90:0/2560293656 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240aaa20 con 0x7f98381624a0 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- 172.21.15.90:0/2560293656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98381624a0 msgr2=0x7f98380d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 --2- 172.21.15.90:0/2560293656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98381624a0 0x7f98380d0d20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f98380b7fb0 tx=0x7f982401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- 172.21.15.90:0/2560293656 shutdown_connections 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 --2- 172.21.15.90:0/2560293656 >> v2:172.21.15.102:3300/0 conn(0x7f98380d2940 0x7f98380b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 --2- 172.21.15.90:0/2560293656 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98381624a0 0x7f98380d0d20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- 172.21.15.90:0/2560293656 >> 172.21.15.90:0/2560293656 conn(0x7f983808bfd0 msgr2=0x7f98380d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- 172.21.15.90:0/2560293656 shutdown_connections 2024-03-31T15:55:29.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- 172.21.15.90:0/2560293656 wait complete. 2024-03-31T15:55:29.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 Processor -- start 2024-03-31T15:55:29.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.152+0000 7f9840200640 1 -- start start 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f983815e320 0x7f9838151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98381624a0 0x7f983806f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98380b35e0 con 0x7f98380d2940 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98380b3460 con 0x7f98381624a0 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f98380b3760 con 0x7f983815e320 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50954/0 (socket says 172.21.15.90:50954) 2024-03-31T15:55:29.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 -- 172.21.15.90:0/691806780 learned_addr learned my addr 172.21.15.90:0/691806780 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983e200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98381624a0 0x7f983806f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983ce00640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f983815e320 0x7f9838151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 -- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f983815e320 msgr2=0x7f9838151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f983815e320 0x7f9838151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 -- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98381624a0 msgr2=0x7f983806f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98381624a0 0x7f983806f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 -- 172.21.15.90:0/691806780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9838152130 con 0x7f98380d2940 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f983d800640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c490 tx=0x7f982c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9835600640 1 -- 172.21.15.90:0/691806780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c017070 con 0x7f98380d2940 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9835600640 1 -- 172.21.15.90:0/691806780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982c003600 con 0x7f98380d2940 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98380c5370 con 0x7f98380d2940 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9835600640 1 -- 172.21.15.90:0/691806780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c003a30 con 0x7f98380d2940 2024-03-31T15:55:29.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9838156c80 con 0x7f98380d2940 2024-03-31T15:55:29.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.156+0000 7f9835600640 1 -- 172.21.15.90:0/691806780 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f982c01a070 con 0x7f98380d2940 2024-03-31T15:55:29.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 Processor -- start 2024-03-31T15:55:29.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f9835600640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 0x7f981806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- start start 2024-03-31T15:55:29.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f9835600640 1 -- 172.21.15.90:0/691806780 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f982c085f40 con 0x7f98380d2940 2024-03-31T15:55:29.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983e200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 0x7f981806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 0x55f008610b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f0086110b0 0x55f008611490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f008463950 con 0x55f00865d4a0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f008400990 con 0x55f00865ddc0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x55f00845f900 con 0x55f0086110b0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50956/0 (socket says 172.21.15.90:50956) 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 -- 172.21.15.90:0/778691864 learned_addr learned my addr 172.21.15.90:0/778691864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983ce00640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 0x55f008610b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983e200640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f0086110b0 0x55f008611490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 -- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f0086110b0 msgr2=0x55f008611490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f0086110b0 0x55f008611490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 -- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 msgr2=0x55f008610b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 0x55f008610b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 -- 172.21.15.90:0/778691864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0084a8100 con 0x55f00865d4a0 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983ce00640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 0x55f008610b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983e200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 0x7f981806a840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9830009390 tx=0x7f9830009650 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f983d800640 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f982c084180 tx=0x7f982c0556e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c673958484a8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f982a000640 1 -- 172.21.15.90:0/778691864 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c06e020 con 0x55f00865d4a0 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f982a000640 1 -- 172.21.15.90:0/778691864 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982c053f80 con 0x55f00865d4a0 2024-03-31T15:55:29.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f982a000640 1 -- 172.21.15.90:0/778691864 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c05e320 con 0x55f00865d4a0 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- 172.21.15.90:0/778691864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 msgr2=0x55f00865d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f982c084180 tx=0x7f982c0556e0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- 172.21.15.90:0/778691864 shutdown_connections 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f0086110b0 0x55f008611490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f00865ddc0 0x55f008610b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 --2- 172.21.15.90:0/778691864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00865d880 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- 172.21.15.90:0/778691864 >> 172.21.15.90:0/778691864 conn(0x55f0084fbb80 msgr2=0x55f008640460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- 172.21.15.90:0/778691864 shutdown_connections 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.160+0000 7f98437a5000 1 -- 172.21.15.90:0/778691864 wait complete. 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- start start 2024-03-31T15:55:29.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 Processor -- start 2024-03-31T15:55:29.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00861de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00861e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f008462c60 con 0x55f00865ddc0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f0084213e0 con 0x55f0086110b0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x55f008470060 con 0x55f00865d4a0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 10 client.0 _ll_register_callbacks cb 0x55f0084c40f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00861de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50968/0 (socket says 172.21.15.90:50968) 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 -- 172.21.15.90:0/547047972 learned_addr learned my addr 172.21.15.90:0/547047972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983ce00640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00861e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 -- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 msgr2=0x55f00861e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00861e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 -- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 msgr2=0x55f00861de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00861de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 -- 172.21.15.90:0/547047972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00842d8c0 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983e200640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9830018e60 tx=0x7f9830013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e576cccb1f26d130 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830027060 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9830022640 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f982c053f80 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f983002d050 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 10 client.5715 fetch_fsmap learned FSMap version 17 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55f0083ebe80 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982c053f80 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f9830030020 con 0x55f00865ddc0 2024-03-31T15:55:29.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 10 client.5715 fetch_fsmap finished waiting for FSMap version 17 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 10 client.5715 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 20 client.5715 populate_metadata read hostname 'smithi090' 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f00861f150 0x55f00866fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 4 client.5715 mds_command: new command op to 5502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 --> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0083febc0 con 0x55f00861f150 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.164+0000 7f983d800640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f00861f150 0x55f00866fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f983d800640 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f00861f150 0x55f00866fa20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0635b34b466ec62 server_cookie=1edeb6ace3dbb653 in_seq=0 out_seq=0 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f981ea00640 10 client.5715 ms_handle_connect on v2:172.21.15.90:6832/353199836 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f981ea00640 1 -- 172.21.15.90:0/547047972 <== mds.0 v2:172.21.15.90:6832/353199836 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0083febc0 con 0x55f00861f150 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f981ea00640 10 client.5715 handle_command_reply: tid=0 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 client.5715 shutdown 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 msgr2=0x55f00861e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f9830018e60 tx=0x7f9830013000 comp rx=0 tx=0).stop 2024-03-31T15:55:29.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 shutdown_connections 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f00861f150 0x55f00866fa20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f00861e880 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00861e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 --2- 172.21.15.90:0/547047972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00861de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 >> 172.21.15.90:0/547047972 conn(0x55f0084fbb80 msgr2=0x55f008610270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f981ea00640 0 client.5715 ms_handle_reset on v2:172.21.15.90:6832/353199836 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 shutdown_connections 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 1 -- 172.21.15.90:0/547047972 wait complete. 2024-03-31T15:55:29.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.168+0000 7f98437a5000 20 client.5715 trim_cache size 0 max 16384 2024-03-31T15:55:29.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.184+0000 7f98437a5000 1 Processor -- start 2024-03-31T15:55:29.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.184+0000 7f98437a5000 1 -- start start 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00863f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00863f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9838207160 con 0x55f00865ddc0 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9838157fa0 con 0x55f0086110b0 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9838206b20 con 0x55f00865d4a0 2024-03-31T15:55:29.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50980/0 (socket says 172.21.15.90:50980) 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 -- 172.21.15.90:0/3496079480 learned_addr learned my addr 172.21.15.90:0/3496079480 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983ce00640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00863f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983d800640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00863f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 -- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 msgr2=0x55f00863f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00863f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 -- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 msgr2=0x55f00863f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00863f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 -- 172.21.15.90:0/3496079480 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f00841e490 con 0x55f00865ddc0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983d800640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00863f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f983e200640 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9830019fd0 tx=0x7f983000d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5ad676ee01a49b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f9829600640 1 -- 172.21.15.90:0/3496079480 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830022640 con 0x55f00865ddc0 2024-03-31T15:55:29.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f9829600640 1 -- 172.21.15.90:0/3496079480 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9830036070 con 0x55f00865ddc0 2024-03-31T15:55:29.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f9829600640 1 -- 172.21.15.90:0/3496079480 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300315f0 con 0x55f00865ddc0 2024-03-31T15:55:29.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- 172.21.15.90:0/3496079480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 msgr2=0x55f0086102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9830019fd0 tx=0x7f983000d040 comp rx=0 tx=0).stop 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- 172.21.15.90:0/3496079480 shutdown_connections 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865ddc0 0x55f0086102e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865d4a0 0x55f00863f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 --2- 172.21.15.90:0/3496079480 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f00863f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- 172.21.15.90:0/3496079480 >> 172.21.15.90:0/3496079480 conn(0x55f0084fbb80 msgr2=0x55f00866aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- 172.21.15.90:0/3496079480 shutdown_connections 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- 172.21.15.90:0/3496079480 wait complete. 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 -- start start 2024-03-31T15:55:29.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.188+0000 7f98437a5000 1 Processor -- start 2024-03-31T15:55:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f008610320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865ddc0 0x55f0086cc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f008418220 con 0x55f00865d4a0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f008467f80 con 0x55f0086110b0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x55f00850dfb0 con 0x55f00865ddc0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 10 client.0 _ll_register_callbacks cb 0x7f98300183f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f008610320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:50996/0 (socket says 172.21.15.90:50996) 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 -- 172.21.15.90:0/845321684 learned_addr learned my addr 172.21.15.90:0/845321684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983e200640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865ddc0 0x55f0086cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 -- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865ddc0 msgr2=0x55f0086cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865ddc0 0x55f0086cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 -- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 msgr2=0x55f008610320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f008610320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 -- 172.21.15.90:0/845321684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00841ce80 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983ce00640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f98240ac040 tx=0x7f982401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba19b024f0c9869 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240aad30 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f98240c0480 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9830036070 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f98240cd050 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 10 client.5721 fetch_fsmap learned FSMap version 17 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55f00840a270 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9830036070 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f98240d0020 con 0x55f00865d4a0 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 10 client.5721 fetch_fsmap finished waiting for FSMap version 17 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 10 client.5721 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-31T15:55:29.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 20 client.5721 populate_metadata read hostname 'smithi090' 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f0086ccf30 0x55f008671270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 4 client.5721 mds_command: new command op to 5502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 --> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] -- command(tid 0: {"prefix": "status"}) -- 0x55f008470e90 con 0x55f0086ccf30 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983d800640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f0086ccf30 0x55f008671270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 10 client.5721 ms_handle_connect on v2:172.21.15.90:6832/353199836 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f983d800640 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f0086ccf30 0x55f008671270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dbeb3e3af2e0c12 server_cookie=4adf4029e25f5d8a in_seq=0 out_seq=0 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 1 -- 172.21.15.90:0/845321684 <== mds.0 v2:172.21.15.90:6832/353199836 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55f008470e90 con 0x55f0086ccf30 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.192+0000 7f981ea00640 10 client.5721 handle_command_reply: tid=0 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 client.5721 shutdown 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 msgr2=0x55f00863fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f98240ac040 tx=0x7f982401ad20 comp rx=0 tx=0).stop 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 shutdown_connections 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x55f00865ddc0 0x55f0086cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:6832/353199836,v1:172.21.15.90:6834/353199836] conn(0x55f0086ccf30 0x55f008671270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f00865d4a0 0x55f00863fb50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 --2- 172.21.15.90:0/845321684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f0086110b0 0x55f008610320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f981ea00640 0 client.5721 ms_handle_reset on v2:172.21.15.90:6832/353199836 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 >> 172.21.15.90:0/845321684 conn(0x55f0084fbb80 msgr2=0x55f00866aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 shutdown_connections 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 1 -- 172.21.15.90:0/845321684 wait complete. 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f98437a5000 20 client.5721 trim_cache size 0 max 16384 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "0135ca30-606a-448f-b61f-3ca22bf9710a", 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-31T15:55:29.194 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5502, 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 10.656435001, 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 17, 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 39, 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 39, 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 18.496881854000002 2024-03-31T15:55:29.195 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 msgr2=0x7f981806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 0x7f981806a840 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9830009390 tx=0x7f9830009650 comp rx=0 tx=0).stop 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 msgr2=0x7f98380cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f982c00c490 tx=0x7f982c001e80 comp rx=0 tx=0).stop 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 shutdown_connections 2024-03-31T15:55:29.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9818068380 0x7f981806a840 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98381624a0 0x7f983806f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f983815e320 0x7f9838151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 --2- 172.21.15.90:0/691806780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98380d2940 0x7f98380cee10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 >> 172.21.15.90:0/691806780 conn(0x7f983808bfd0 msgr2=0x7f98380b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 shutdown_connections 2024-03-31T15:55:29.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.196+0000 7f9840200640 1 -- 172.21.15.90:0/691806780 wait complete. 2024-03-31T15:55:29.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:55:29.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 Processor -- start 2024-03-31T15:55:29.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 -- start start 2024-03-31T15:55:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 0x7f936015fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 0x7f93600b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93600b2c20 0x7f93600cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9360160bf0 con 0x7f93600b2c20 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9360160350 con 0x7f93600d2680 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9366000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9360160630 con 0x7f936015c120 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93600b2c20 0x7f93600cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93600b2c20 0x7f93600cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51002/0 (socket says 172.21.15.90:51002) 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 -- 172.21.15.90:0/814085597 learned_addr learned my addr 172.21.15.90:0/814085597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935f400640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 0x7f936015fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9357e00640 1 --2- 172.21.15.90:0/814085597 >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 0x7f93600b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 -- 172.21.15.90:0/814085597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 msgr2=0x7f936015fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 0x7f936015fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 -- 172.21.15.90:0/814085597 >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 msgr2=0x7f93600b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 --2- 172.21.15.90:0/814085597 >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 0x7f93600b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935fe00640 1 -- 172.21.15.90:0/814085597 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9360160910 con 0x7f93600b2c20 2024-03-31T15:55:29.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f935f400640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 0x7f936015fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:29.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.356+0000 7f9357e00640 1 --2- 172.21.15.90:0/814085597 >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 0x7f93600b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:29.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f935fe00640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600b2c20 0x7f93600cca30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f93600b4ab0 tx=0x7f935801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da32042caa9bac50 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f935ea00640 1 -- 172.21.15.90:0/814085597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580a5630 con 0x7f93600b2c20 2024-03-31T15:55:29.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f935ea00640 1 -- 172.21.15.90:0/814085597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93580a57d0 con 0x7f93600b2c20 2024-03-31T15:55:29.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f935ea00640 1 -- 172.21.15.90:0/814085597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580a5ba0 con 0x7f93600b2c20 2024-03-31T15:55:29.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 -- 172.21.15.90:0/814085597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600b2c20 msgr2=0x7f93600cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600b2c20 0x7f93600cca30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f93600b4ab0 tx=0x7f935801c8a0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 -- 172.21.15.90:0/814085597 shutdown_connections 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93600b2c20 0x7f93600cca30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 --2- 172.21.15.90:0/814085597 >> v2:172.21.15.102:3300/0 conn(0x7f93600d2680 0x7f93600b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 --2- 172.21.15.90:0/814085597 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f936015c120 0x7f936015fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 -- 172.21.15.90:0/814085597 >> 172.21.15.90:0/814085597 conn(0x7f936009aa40 msgr2=0x7f936009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 -- 172.21.15.90:0/814085597 shutdown_connections 2024-03-31T15:55:29.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.360+0000 7f9366000640 1 -- 172.21.15.90:0/814085597 wait complete. 2024-03-31T15:55:29.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 Processor -- start 2024-03-31T15:55:29.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- start start 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600b2c20 0x7f9360157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93600d2680 0x7f936015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f93600cea90 con 0x7f936015c120 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f93600ce910 con 0x7f93600b2c20 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f93600cec10 con 0x7f93600d2680 2024-03-31T15:55:29.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51004/0 (socket says 172.21.15.90:51004) 2024-03-31T15:55:29.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 -- 172.21.15.90:0/80518540 learned_addr learned my addr 172.21.15.90:0/80518540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:29.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9357e00640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93600d2680 0x7f936015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935f400640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600b2c20 0x7f9360157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 -- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93600d2680 msgr2=0x7f936015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93600d2680 0x7f936015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 -- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600b2c20 msgr2=0x7f9360157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600b2c20 0x7f9360157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 -- 172.21.15.90:0/80518540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93600c6e30 con 0x7f936015c120 2024-03-31T15:55:29.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f935fe00640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f935801d140 tx=0x7f9358019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580bc020 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93580b8670 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93600c70d0 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358009510 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.364+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93600c7690 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.368+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93600686d0 con 0x7f936015c120 2024-03-31T15:55:29.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.368+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f93580bf020 con 0x7f936015c120 2024-03-31T15:55:29.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.368+0000 7f9357400640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 0x7f933c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:29.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.368+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9358121af0 con 0x7f936015c120 2024-03-31T15:55:29.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.368+0000 7f935f400640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 0x7f933c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:29.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.372+0000 7f935f400640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 0x7f933c06a840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f93500045f0 tx=0x7f9350008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:29.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.372+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f93580c6030 con 0x7f936015c120 2024-03-31T15:55:29.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.564+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f93600b1b60 con 0x7f936015c120 2024-03-31T15:55:29.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.564+0000 7f9357400640 1 -- 172.21.15.90:0/80518540 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f93580165e0 con 0x7f936015c120 2024-03-31T15:55:29.560 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 msgr2=0x7f933c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 0x7f933c06a840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f93500045f0 tx=0x7f9350008040 comp rx=0 tx=0).stop 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 msgr2=0x7f936015afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f935801d140 tx=0x7f9358019640 comp rx=0 tx=0).stop 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 shutdown_connections 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f933c068380 0x7f933c06a840 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f936015c120 0x7f936015afb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93600d2680 0x7f936015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 --2- 172.21.15.90:0/80518540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93600b2c20 0x7f9360157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 >> 172.21.15.90:0/80518540 conn(0x7f936009aa40 msgr2=0x7f9360070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 shutdown_connections 2024-03-31T15:55:29.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29.568+0000 7f9366000640 1 -- 172.21.15.90:0/80518540 wait complete. 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:55:29.581 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:29.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:29.582 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.592 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42394sec preferred_lft 42394sec 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:55:29.593 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:29.594 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:55:29.595 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:29.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:29.595 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:55:29.649 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:55:29.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:29.651 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:55:29.703 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:55:29.703 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:55:29.703 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:55:29.703 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:29.704 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:55:29.704 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:55:29.704 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:55:29.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:55:29.704 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:55:29.705 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:55:29.705 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:55:29.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:55:29.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:29.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:29.837 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:55:29.846 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:55:29.846 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:55:29.847 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:29.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:29.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:55:29.908 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:55:29.908 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:29.908 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:29.960 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:29.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:29.960 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:30.013 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:30.077 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:30.129 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:30.129 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:30.129 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:30.129 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:30.129 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:30.130 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:30.131 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:30.132 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:30.133 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:30.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:30.135 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:30.184 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:55:30.185 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:55:30.230 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:30.250 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:30.301 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:30.301 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:30.301 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:30.301 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:30.301 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:30.302 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:30.303 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:30.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:30.304 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:30.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:55:30.332+0000 7ff1d5801580 -1 init, newargv = 0x562d898382b0 newargc=15 2024-03-31T15:55:30.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[38840]: starting ceph client 2024-03-31T15:55:30.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[38840]: starting fuse 2024-03-31T15:55:30.333 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:55:30.333 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:30.334 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:55:30.335 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:55:30.457 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.38840.asok 2024-03-31T15:55:30.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:30.468 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38840.asok 2024-03-31T15:55:30.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:30.468 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38840.asok status 2024-03-31T15:55:30.641 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "38840", 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:55:30.642 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5733, 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 5733 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2083982098 2024-03-31T15:55:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2083982098 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.5733 192.168.0.1:0/2083982098", 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/2083982098", 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 17, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 39, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:55:30.645 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:55:30.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:30.656 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:55:30.656 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:30.665 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:55:30.665 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:30.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:30.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:30.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-31T15:55:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 Processor -- start 2024-03-31T15:55:30.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 -- start start 2024-03-31T15:55:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f966015c340 0x7f96600cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f96600ced10 0x7f96600ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96600b25d0 con 0x7f96600b1e10 2024-03-31T15:55:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f96600b31e0 con 0x7f96600ced10 2024-03-31T15:55:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.912+0000 7f9668600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f96600b28b0 con 0x7f966015c340 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9665200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f96600b1e10 0x7f96600ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44326/0 (socket says 172.21.15.90:44326) 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9665200640 1 -- 172.21.15.90:0/1377201003 learned_addr learned my addr 172.21.15.90:0/1377201003 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 --2- 172.21.15.90:0/1377201003 >> v2:172.21.15.90:3300/0 conn(0x7f96600ced10 0x7f96600ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9665c00640 1 --2- 172.21.15.90:0/1377201003 >> v2:172.21.15.102:3300/0 conn(0x7f966015c340 0x7f96600cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 -- 172.21.15.90:0/1377201003 >> v2:172.21.15.102:3300/0 conn(0x7f966015c340 msgr2=0x7f96600cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 --2- 172.21.15.90:0/1377201003 >> v2:172.21.15.102:3300/0 conn(0x7f966015c340 0x7f96600cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 -- 172.21.15.90:0/1377201003 >> v1:172.21.15.90:6789/0 conn(0x7f96600b1e10 legacy=0x7f96600ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 -- 172.21.15.90:0/1377201003 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96600b2b90 con 0x7f96600ced10 2024-03-31T15:55:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f965fe00640 1 -- 172.21.15.90:0/1377201003 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906497169 0 0) 0x7f96600b25d0 con 0x7f96600b1e10 2024-03-31T15:55:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f9666600640 1 --2- 172.21.15.90:0/1377201003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96600ced10 0x7f96600ccd00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f966009ba10 tx=0x7f965801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5cd1a9c61eb42f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f965fe00640 1 -- 172.21.15.90:0/1377201003 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580b2070 con 0x7f96600ced10 2024-03-31T15:55:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f965fe00640 1 -- 172.21.15.90:0/1377201003 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96580a58a0 con 0x7f96600ced10 2024-03-31T15:55:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.916+0000 7f965fe00640 1 -- 172.21.15.90:0/1377201003 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580a5c70 con 0x7f96600ced10 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- 172.21.15.90:0/1377201003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96600ced10 msgr2=0x7f96600ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- 172.21.15.90:0/1377201003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96600ced10 0x7f96600ccd00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f966009ba10 tx=0x7f965801c930 comp rx=0 tx=0).stop 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- 172.21.15.90:0/1377201003 shutdown_connections 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- 172.21.15.90:0/1377201003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96600ced10 0x7f96600ccd00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- 172.21.15.90:0/1377201003 >> v2:172.21.15.102:3300/0 conn(0x7f966015c340 0x7f96600cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- 172.21.15.90:0/1377201003 >> 172.21.15.90:0/1377201003 conn(0x7f966009aa40 msgr2=0x7f966009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- 172.21.15.90:0/1377201003 shutdown_connections 2024-03-31T15:55:30.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- 172.21.15.90:0/1377201003 wait complete. 2024-03-31T15:55:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 Processor -- start 2024-03-31T15:55:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- start start 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96600b1e10 0x7f966015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96600ced10 0x7f96600caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f96600aed00 con 0x7f966015c340 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f96600aeb80 con 0x7f96600b1e10 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9668600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9660168ee0 con 0x7f96600ced10 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51022/0 (socket says 172.21.15.90:51022) 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 -- 172.21.15.90:0/92523699 learned_addr learned my addr 172.21.15.90:0/92523699 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9665c00640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96600b1e10 0x7f966015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9665200640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96600ced10 0x7f96600caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 -- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96600ced10 msgr2=0x7f96600caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96600ced10 0x7f96600caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 -- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96600b1e10 msgr2=0x7f966015b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96600b1e10 0x7f966015b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.920+0000 7f9666600640 1 -- 172.21.15.90:0/92523699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96601599d0 con 0x7f966015c340 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f9666600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f96580b0040 tx=0x7f96580196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580b2050 con 0x7f966015c340 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96580be070 con 0x7f966015c340 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9660159cb0 con 0x7f966015c340 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580ada90 con 0x7f966015c340 2024-03-31T15:55:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966015a270 con 0x7f966015c340 2024-03-31T15:55:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96600686d0 con 0x7f966015c340 2024-03-31T15:55:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.924+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f96580a5720 con 0x7f966015c340 2024-03-31T15:55:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.928+0000 7f965d600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 0x7f964006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.928+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f9658122500 con 0x7f966015c340 2024-03-31T15:55:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.928+0000 7f9665c00640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 0x7f964006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.928+0000 7f9665c00640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 0x7f964006a840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f96600afc00 tx=0x7f9654009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:30.928+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9658016520 con 0x7f966015c340 2024-03-31T15:55:31.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.084+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7f96600b0d70 con 0x7f966015c340 2024-03-31T15:55:31.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.604+0000 7f965d600640 1 -- 172.21.15.90:0/92523699 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v41) ==== 124+0+0 (secure 0 0 0) 0x7f96580f2430 con 0x7f966015c340 2024-03-31T15:55:31.599 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'data_new' created 2024-03-31T15:55:31.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 msgr2=0x7f964006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 0x7f964006a840 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f96600afc00 tx=0x7f9654009190 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 msgr2=0x7f96600aef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f96580b0040 tx=0x7f96580196d0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 shutdown_connections 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9640068380 0x7f964006a840 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f966015c340 0x7f96600aef20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f96600ced10 0x7f96600caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 --2- 172.21.15.90:0/92523699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96600b1e10 0x7f966015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 >> 172.21.15.90:0/92523699 conn(0x7f966009aa40 msgr2=0x7f9660155670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 shutdown_connections 2024-03-31T15:55:31.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.608+0000 7f9668600640 1 -- 172.21.15.90:0/92523699 wait complete. 2024-03-31T15:55:31.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-31T15:55:31.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 Processor -- start 2024-03-31T15:55:31.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 -- start start 2024-03-31T15:55:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f36600d2b20 0x7f36600cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 0x7f36600af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36600cefc0 con 0x7f366015c320 2024-03-31T15:55:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f36600aed70 con 0x7f36600d2b20 2024-03-31T15:55:31.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3666a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f36600cf2a0 con 0x7f36600cd640 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365f400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f36600d2b20 0x7f36600cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f366015c320 0x7f36600cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44332/0 (socket says 172.21.15.90:44332) 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365fe00640 1 -- 172.21.15.90:0/2024595519 learned_addr learned my addr 172.21.15.90:0/2024595519 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3664a00640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 0x7f36600af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365f400640 1 -- 172.21.15.90:0/2024595519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 msgr2=0x7f36600af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:31.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365f400640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 0x7f36600af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365f400640 1 -- 172.21.15.90:0/2024595519 >> v1:172.21.15.90:6789/0 conn(0x7f366015c320 legacy=0x7f36600cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365f400640 1 -- 172.21.15.90:0/2024595519 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36600cf580 con 0x7f36600d2b20 2024-03-31T15:55:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f365ea00640 1 -- 172.21.15.90:0/2024595519 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2060413188 0 0) 0x7f36600cefc0 con 0x7f366015c320 2024-03-31T15:55:31.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.776+0000 7f3664a00640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 0x7f36600af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f365f400640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36600cd100 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097c0 tx=0x7f364c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1844d32a3bf6b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f365ea00640 1 -- 172.21.15.90:0/2024595519 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c0b0070 con 0x7f36600d2b20 2024-03-31T15:55:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f365ea00640 1 -- 172.21.15.90:0/2024595519 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36600d0f20 con 0x7f36600d2b20 2024-03-31T15:55:31.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f365ea00640 1 -- 172.21.15.90:0/2024595519 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c0ab820 con 0x7f36600d2b20 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 -- 172.21.15.90:0/2024595519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 msgr2=0x7f36600cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36600cd100 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097c0 tx=0x7f364c01c410 comp rx=0 tx=0).stop 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 -- 172.21.15.90:0/2024595519 shutdown_connections 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36600cd640 0x7f36600af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 --2- 172.21.15.90:0/2024595519 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36600cd100 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 -- 172.21.15.90:0/2024595519 >> 172.21.15.90:0/2024595519 conn(0x7f36600d1f20 msgr2=0x7f36600d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 -- 172.21.15.90:0/2024595519 shutdown_connections 2024-03-31T15:55:31.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.780+0000 7f3666a00640 1 -- 172.21.15.90:0/2024595519 wait complete. 2024-03-31T15:55:31.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 Processor -- start 2024-03-31T15:55:31.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 -- start start 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36600cd640 0x7f366015d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f366015c320 0x7f36600ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36600b0320 con 0x7f36600d2b20 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36600b01a0 con 0x7f36600cd640 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f36600b04a0 con 0x7f366015c320 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51032/0 (socket says 172.21.15.90:51032) 2024-03-31T15:55:31.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 -- 172.21.15.90:0/1343780006 learned_addr learned my addr 172.21.15.90:0/1343780006 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3664a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f366015c320 0x7f36600ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365fe00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36600cd640 0x7f366015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 -- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f366015c320 msgr2=0x7f36600ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f366015c320 0x7f36600ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 -- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36600cd640 msgr2=0x7f366015d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36600cd640 0x7f366015d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 -- 172.21.15.90:0/1343780006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3660153270 con 0x7f36600d2b20 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f365f400640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f364c0ae040 tx=0x7f364c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c0b0050 con 0x7f36600d2b20 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f364c0bc070 con 0x7f36600d2b20 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3660153550 con 0x7f36600d2b20 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.784+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c0b78b0 con 0x7f36600d2b20 2024-03-31T15:55:31.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3660153b10 con 0x7f36600d2b20 2024-03-31T15:55:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36600685f0 con 0x7f36600d2b20 2024-03-31T15:55:31.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f364c0b7a50 con 0x7f36600d2b20 2024-03-31T15:55:31.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f3657e00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 0x7f364006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:31.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6493+0+0 (secure 0 0 0) 0x7f364c1214c0 con 0x7f36600d2b20 2024-03-31T15:55:31.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.788+0000 7f365fe00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 0x7f364006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:31.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.792+0000 7f365fe00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 0x7f364006a840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3650007fd0 tx=0x7f3650007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:31.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.792+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f364c016090 con 0x7f36600d2b20 2024-03-31T15:55:31.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:31.988+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7f36600b2770 con 0x7f36600d2b20 2024-03-31T15:55:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.624+0000 7f3657e00640 1 -- 172.21.15.90:0/1343780006 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v18) ==== 131+0+0 (secure 0 0 0) 0x7f364c0f0f70 con 0x7f36600d2b20 2024-03-31T15:55:32.618 INFO:teuthology.orchestra.run.smithi090.stderr:added data pool 7 to fsmap 2024-03-31T15:55:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 msgr2=0x7f364006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 0x7f364006a840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3650007fd0 tx=0x7f3650007d60 comp rx=0 tx=0).stop 2024-03-31T15:55:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 msgr2=0x7f36601524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f364c0ae040 tx=0x7f364c01ac50 comp rx=0 tx=0).stop 2024-03-31T15:55:32.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 shutdown_connections 2024-03-31T15:55:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.628+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3640068380 0x7f364006a840 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f366015c320 0x7f36600ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f36600d2b20 0x7f36601524d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 --2- 172.21.15.90:0/1343780006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36600cd640 0x7f366015d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 >> 172.21.15.90:0/1343780006 conn(0x7f36600d1f20 msgr2=0x7f36600cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 shutdown_connections 2024-03-31T15:55:32.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.632+0000 7f3666a00640 1 -- 172.21.15.90:0/1343780006 wait complete. 2024-03-31T15:55:32.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:55:32.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.788+0000 7f589f800640 1 Processor -- start 2024-03-31T15:55:32.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.788+0000 7f589f800640 1 -- start start 2024-03-31T15:55:32.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589f800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589f800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58980b3500 0x7f58980ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589f800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58980cea80 con 0x7f58980c4ef0 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589f800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f58980d2c80 con 0x7f58980b3500 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589f800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f58980b2770 con 0x7f58980b3080 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589d800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f58980b3500 0x7f58980ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589ce00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f58980c4ef0 0x7f58980c52d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:44338/0 (socket says 172.21.15.90:44338) 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589ce00640 1 -- 172.21.15.90:0/1113082878 learned_addr learned my addr 172.21.15.90:0/1113082878 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f588fe00640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589d800640 1 -- 172.21.15.90:0/1113082878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 msgr2=0x7f58980cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589d800640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589d800640 1 -- 172.21.15.90:0/1113082878 >> v1:172.21.15.90:6789/0 conn(0x7f58980c4ef0 legacy=0x7f58980c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f589d800640 1 -- 172.21.15.90:0/1113082878 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980d2070 con 0x7f58980b3500 2024-03-31T15:55:32.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.792+0000 7f5897e00640 1 -- 172.21.15.90:0/1113082878 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293851456 0 0) 0x7f58980cea80 con 0x7f58980c4ef0 2024-03-31T15:55:32.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589d800640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f58980ccc20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5890009d90 tx=0x7f589001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e0dafb77116d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f5897e00640 1 -- 172.21.15.90:0/1113082878 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58900b2070 con 0x7f58980b3500 2024-03-31T15:55:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f5897e00640 1 -- 172.21.15.90:0/1113082878 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58900a5920 con 0x7f58980b3500 2024-03-31T15:55:32.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f5897e00640 1 -- 172.21.15.90:0/1113082878 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58900a5cf0 con 0x7f58980b3500 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- 172.21.15.90:0/1113082878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 msgr2=0x7f58980ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f58980ccc20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5890009d90 tx=0x7f589001c970 comp rx=0 tx=0).stop 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- 172.21.15.90:0/1113082878 shutdown_connections 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f58980ccc20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- 172.21.15.90:0/1113082878 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- 172.21.15.90:0/1113082878 >> 172.21.15.90:0/1113082878 conn(0x7f589809aa40 msgr2=0x7f589809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- 172.21.15.90:0/1113082878 shutdown_connections 2024-03-31T15:55:32.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- 172.21.15.90:0/1113082878 wait complete. 2024-03-31T15:55:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 Processor -- start 2024-03-31T15:55:32.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- start start 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58980c4ef0 0x7f58980b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f589816f730 con 0x7f58980b3500 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f589816f5b0 con 0x7f58980c4ef0 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.796+0000 7f589f800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f589816f8b0 con 0x7f58980b3080 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f589ce00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51046/0 (socket says 172.21.15.90:51046) 2024-03-31T15:55:32.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 -- 172.21.15.90:0/1847946110 learned_addr learned my addr 172.21.15.90:0/1847946110 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f589d800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58980c4ef0 0x7f58980b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 -- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 msgr2=0x7f58980b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 -- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58980c4ef0 msgr2=0x7f58980b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58980c4ef0 0x7f58980b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 -- 172.21.15.90:0/1847946110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980aebb0 con 0x7f58980b3500 2024-03-31T15:55:32.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f588fe00640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f588000d5f0 tx=0x7f588000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800125f0 con 0x7f58980b3500 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880019070 con 0x7f58980b3500 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58980b5af0 con 0x7f58980b3500 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800129b0 con 0x7f58980b3500 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58980b6390 con 0x7f58980b3500 2024-03-31T15:55:32.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58980686d0 con 0x7f58980b3500 2024-03-31T15:55:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.800+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5880012b50 con 0x7f58980b3500 2024-03-31T15:55:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.804+0000 7f5895600640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 0x7f587406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:32.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.804+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6525+0+0 (secure 0 0 0) 0x7f5880086fc0 con 0x7f58980b3500 2024-03-31T15:55:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.804+0000 7f589ce00640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 0x7f587406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:32.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.804+0000 7f589ce00640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 0x7f587406a840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5888007fd0 tx=0x7f5888007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:32.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:32.812+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5880056bf0 con 0x7f58980b3500 2024-03-31T15:55:32.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.000+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58980cefb0 con 0x7f58980b3500 2024-03-31T15:55:32.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f5895600640 1 -- 172.21.15.90:0/1847946110 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5186 (secure 0 0 0) 0x7f588005aef0 con 0x7f58980b3500 2024-03-31T15:55:32.997 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:32.997 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:31.990711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:55:32.997 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 msgr2=0x7f587406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 0x7f587406a840 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5888007fd0 tx=0x7f5888007d60 comp rx=0 tx=0).stop 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 msgr2=0x7f589814d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f588000d5f0 tx=0x7f588000dab0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 shutdown_connections 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5874068380 0x7f587406a840 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58980c4ef0 0x7f58980b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58980b3500 0x7f589814d230 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:32.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 --2- 172.21.15.90:0/1847946110 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58980b3080 0x7f58980b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 >> 172.21.15.90:0/1847946110 conn(0x7f589809aa40 msgr2=0x7f58980cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:33.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 shutdown_connections 2024-03-31T15:55:33.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.004+0000 7f589f800640 1 -- 172.21.15.90:0/1847946110 wait complete. 2024-03-31T15:55:33.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:33.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 Processor -- start 2024-03-31T15:55:33.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 -- start start 2024-03-31T15:55:33.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 0x7f025c0adf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f025c06d210 0x7f025c0ae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f025c15fbd0 con 0x7f025c06e3e0 2024-03-31T15:55:33.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f025c15feb0 con 0x7f025c06d210 2024-03-31T15:55:33.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0262a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f025c160190 con 0x7f025c15c020 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0260a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f025c06d210 0x7f025c0ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0253e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f025c06e3e0 0x7f025c06cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41680/0 (socket says 172.21.15.90:41680) 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0253e00640 1 -- 172.21.15.90:0/2128861329 learned_addr learned my addr 172.21.15.90:0/2128861329 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f025be00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 0x7f025c0adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0260a00640 1 -- 172.21.15.90:0/2128861329 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 msgr2=0x7f025c0adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0260a00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 0x7f025c0adf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0260a00640 1 -- 172.21.15.90:0/2128861329 >> v1:172.21.15.90:6789/0 conn(0x7f025c06e3e0 legacy=0x7f025c06cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f0260a00640 1 -- 172.21.15.90:0/2128861329 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c160470 con 0x7f025c06d210 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f025be00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 0x7f025c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:33.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.160+0000 7f025b400640 1 -- 172.21.15.90:0/2128861329 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897586580 0 0) 0x7f025c15fbd0 con 0x7f025c06e3e0 2024-03-31T15:55:33.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0260a00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c06d210 0x7f025c0ae480 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0254009d90 tx=0x7f025401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0710edddb4ca859 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f025b400640 1 -- 172.21.15.90:0/2128861329 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02540b2070 con 0x7f025c06d210 2024-03-31T15:55:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f025b400640 1 -- 172.21.15.90:0/2128861329 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02540a5930 con 0x7f025c06d210 2024-03-31T15:55:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f025b400640 1 -- 172.21.15.90:0/2128861329 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02540a5d00 con 0x7f025c06d210 2024-03-31T15:55:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 -- 172.21.15.90:0/2128861329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c06d210 msgr2=0x7f025c0ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c06d210 0x7f025c0ae480 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0254009d90 tx=0x7f025401c8b0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 -- 172.21.15.90:0/2128861329 shutdown_connections 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c06d210 0x7f025c0ae480 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 --2- 172.21.15.90:0/2128861329 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c15c020 0x7f025c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 -- 172.21.15.90:0/2128861329 >> 172.21.15.90:0/2128861329 conn(0x7f025c0d13a0 msgr2=0x7f025c0d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 -- 172.21.15.90:0/2128861329 shutdown_connections 2024-03-31T15:55:33.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.164+0000 7f0262a00640 1 -- 172.21.15.90:0/2128861329 wait complete. 2024-03-31T15:55:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 Processor -- start 2024-03-31T15:55:33.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- start start 2024-03-31T15:55:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c06d210 0x7f025c0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c06e3e0 0x7f025c0c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f025c0cc270 con 0x7f025c15c020 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f025c0cc0f0 con 0x7f025c06e3e0 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f025c0cc3f0 con 0x7f025c06d210 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54446/0 (socket says 172.21.15.90:54446) 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f025be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c06d210 0x7f025c0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0253e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c06e3e0 0x7f025c0c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 -- 172.21.15.90:0/3707028637 learned_addr learned my addr 172.21.15.90:0/3707028637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 -- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c06d210 msgr2=0x7f025c0c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c06d210 0x7f025c0c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 -- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c06e3e0 msgr2=0x7f025c0c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c06e3e0 0x7f025c0c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 -- 172.21.15.90:0/3707028637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025c0ca920 con 0x7f025c15c020 2024-03-31T15:55:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0260a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f02540b0040 tx=0x7f0254019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02540b2050 con 0x7f025c15c020 2024-03-31T15:55:33.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02540be070 con 0x7f025c15c020 2024-03-31T15:55:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02540adb10 con 0x7f025c15c020 2024-03-31T15:55:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025c0cac00 con 0x7f025c15c020 2024-03-31T15:55:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.168+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f025c0c74e0 con 0x7f025c15c020 2024-03-31T15:55:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f025c0686d0 con 0x7f025c15c020 2024-03-31T15:55:33.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f02540adcb0 con 0x7f025c15c020 2024-03-31T15:55:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f0258c00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 0x7f023806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6525+0+0 (secure 0 0 0) 0x7f0254121ee0 con 0x7f025c15c020 2024-03-31T15:55:33.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f025be00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 0x7f023806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.172+0000 7f025be00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 0x7f023806a870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f024c0045f0 tx=0x7f024c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.180+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0254016560 con 0x7f025c15c020 2024-03-31T15:55:33.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.340+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f025c0cadd0 con 0x7f025c15c020 2024-03-31T15:55:33.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.340+0000 7f0258c00640 1 -- 172.21.15.90:0/3707028637 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18780 (secure 0 0 0) 0x7f02540f1c30 con 0x7f025c15c020 2024-03-31T15:55:33.335 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:33.335 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":42,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:31.990477+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 msgr2=0x7f023806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 0x7f023806a870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f024c0045f0 tx=0x7f024c008040 comp rx=0 tx=0).stop 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 msgr2=0x7f025c1552a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f02540b0040 tx=0x7f0254019650 comp rx=0 tx=0).stop 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 shutdown_connections 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f02380683b0 0x7f023806a870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f025c15c020 0x7f025c1552a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f025c06e3e0 0x7f025c0c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 --2- 172.21.15.90:0/3707028637 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f025c06d210 0x7f025c0c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 >> 172.21.15.90:0/3707028637 conn(0x7f025c0d13a0 msgr2=0x7f025c1582e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:33.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 shutdown_connections 2024-03-31T15:55:33.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.344+0000 7f0262a00640 1 -- 172.21.15.90:0/3707028637 wait complete. 2024-03-31T15:55:33.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:33.357 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-31T15:55:33.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:33.374 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-31T15:55:33.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-31T15:55:33.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 Processor -- start 2024-03-31T15:55:33.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 -- start start 2024-03-31T15:55:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1c1c15e4e0 0x7f1c1c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1c15e8c0 con 0x7f1c1c0ce7a0 2024-03-31T15:55:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1c0d0ff0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c22400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d12d0 con 0x7f1c1c06ce50 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c1b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1c1c15e4e0 0x7f1c1c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c1b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1c1c15e4e0 0x7f1c1c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:35078/0 (socket says 172.21.15.90:35078) 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c06d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:46734/0 (socket says 172.21.15.90:46734) 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.644+0000 7f1c1b400640 1 -- 172.21.15.90:0/892022983 learned_addr learned my addr 172.21.15.90:0/892022983 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1b400640 1 -- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 msgr2=0x7f1c1c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1b400640 1 --2- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1b400640 1 -- 172.21.15.90:0/892022983 >> v1:172.21.15.102:6789/0 conn(0x7f1c1c0ce7a0 legacy=0x7f1c1c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1b400640 1 -- 172.21.15.90:0/892022983 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c0cc710 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1b400640 1 --2- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c06c910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009870 tx=0x7f1c0c01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cc1b2708e4dd73a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1a000640 1 -- 172.21.15.90:0/892022983 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0b0070 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1a000640 1 -- 172.21.15.90:0/892022983 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0ab600 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c1a000640 1 -- 172.21.15.90:0/892022983 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0ab9d0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 -- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c15e4e0 msgr2=0x7f1c1c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 --2- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c06c910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009870 tx=0x7f1c0c01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 -- 172.21.15.90:0/892022983 shutdown_connections 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 --2- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 --2- 172.21.15.90:0/892022983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c06c910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 -- 172.21.15.90:0/892022983 >> 172.21.15.90:0/892022983 conn(0x7f1c1c0d1f20 msgr2=0x7f1c1c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 -- 172.21.15.90:0/892022983 shutdown_connections 2024-03-31T15:55:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.648+0000 7f1c22400640 1 -- 172.21.15.90:0/892022983 wait complete. 2024-03-31T15:55:33.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.652+0000 7f1c22400640 1 Processor -- start 2024-03-31T15:55:33.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.652+0000 7f1c22400640 1 -- start start 2024-03-31T15:55:33.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c0ce7a0 0x7f1c1c149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d0d90 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d0c10 con 0x7f1c1c0ce7a0 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c16b840 con 0x7f1c1c06ce50 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54458/0 (socket says 172.21.15.90:54458) 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 -- 172.21.15.90:0/2782146496 learned_addr learned my addr 172.21.15.90:0/2782146496 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1aa00640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c0ce7a0 0x7f1c1c149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1b400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 -- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 msgr2=0x7f1c1c155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 -- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c0ce7a0 msgr2=0x7f1c1c149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c0ce7a0 0x7f1c1c149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 -- 172.21.15.90:0/2782146496 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c0c6810 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c1be00640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c7a0 tx=0x7f1c1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14003b80 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c14003d20 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c1564b0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400aa90 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c155ef0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c1c0685f0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.656+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1c1400ac30 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.660+0000 7f1c13400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 0x7f1bfc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:33.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.660+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f1c14086ad0 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.660+0000 7f1c1b400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 0x7f1bfc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:33.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.660+0000 7f1c1b400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 0x7f1bfc06a840 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ad000 tx=0x7f1c0c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:33.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.668+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1c14056700 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.884+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7f1c1c157840 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.888+0000 7f1c13400640 1 -- 172.21.15.90:0/2782146496 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v14) ==== 210+0+167 (secure 0 0 0) 0x7f1c1405aa00 con 0x7f1c1c15e4e0 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow rw" 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-31T15:55:33.882 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.888+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 msgr2=0x7f1bfc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.888+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 0x7f1bfc06a840 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ad000 tx=0x7f1c0c008040 comp rx=0 tx=0).stop 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.888+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 msgr2=0x7f1c1c15bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.888+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400c7a0 tx=0x7f1c1400cc60 comp rx=0 tx=0).stop 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 shutdown_connections 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1bfc068380 0x7f1bfc06a840 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c1c15e4e0 0x7f1c1c15bff0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c1c0ce7a0 0x7f1c1c149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 --2- 172.21.15.90:0/2782146496 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1c1c06ce50 0x7f1c1c155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 >> 172.21.15.90:0/2782146496 conn(0x7f1c1c0d1f20 msgr2=0x7f1c1c0c6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 shutdown_connections 2024-03-31T15:55:33.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:33.892+0000 7f1c22400640 1 -- 172.21.15.90:0/2782146496 wait complete. 2024-03-31T15:55:33.903 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:55:33.904 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:55:33.913 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:34.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.096+0000 7f7b9e400640 1 Processor -- start 2024-03-31T15:55:34.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.096+0000 7f7b9e400640 1 -- start start 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b9e400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b9815beb0 0x7f7b9815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b9e400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98162170 con 0x7f7b9806bf30 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b9e400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98162450 con 0x7f7b9806edb0 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b9e400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7b98162730 con 0x7f7b9815beb0 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7b9806bf30 0x7f7b980715e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41690/0 (socket says 172.21.15.90:41690) 2024-03-31T15:55:34.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97e00640 1 -- 172.21.15.90:0/2253332439 learned_addr learned my addr 172.21.15.90:0/2253332439 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266721290 0 0) 0x7f7b98162170 con 0x7f7b9806bf30 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b7c003530 con 0x7f7b9806bf30 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97400640 1 --2- 172.21.15.90:0/2253332439 >> v2:172.21.15.102:3300/0 conn(0x7f7b9815beb0 0x7f7b9815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3572355445 0 0) 0x7f7b7c003530 con 0x7f7b9806bf30 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 >> v2:172.21.15.102:3300/0 conn(0x7f7b9815beb0 msgr2=0x7f7b9815f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 --2- 172.21.15.90:0/2253332439 >> v2:172.21.15.102:3300/0 conn(0x7f7b9815beb0 0x7f7b9815f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 >> v1:172.21.15.102:6789/0 conn(0x7f7b9806edb0 legacy=0x7f7b9806b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b98162a10 con 0x7f7b9806bf30 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7b90001a20 con 0x7f7b9806bf30 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 >> v1:172.21.15.90:6789/0 conn(0x7f7b9806bf30 legacy=0x7f7b980715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0039c0 0x7f7b7c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b7c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b7c0053d0 0x7f7b7c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b98162730 con 0x7f7b7c004810 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7c00c260 con 0x7f7b7c0039c0 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7c00c460 con 0x7f7b7c0053d0 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97400640 1 --2- 172.21.15.90:0/2253332439 >> v2:172.21.15.102:3300/0 conn(0x7f7b9815beb0 0x7f7b9815f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b7c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97e00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b7c0053d0 0x7f7b7c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97400640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0039c0 0x7f7b7c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 -- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b7c0053d0 msgr2=0x7f7b7c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b7c0053d0 0x7f7b7c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 -- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0039c0 msgr2=0x7f7b7c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b7c0039c0 0x7f7b7c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b8fe00640 1 -- 172.21.15.90:0/2253332439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7c003530 con 0x7f7b7c004810 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.100+0000 7f7b97e00640 1 -- 172.21.15.90:0/2253332439 reap_dead start 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b7c004c90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7b80001780 tx=0x7f7b8001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ab622514c40b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b800013a0 con 0x7f7b7c004810 2024-03-31T15:55:34.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b800052f0 con 0x7f7b7c004810 2024-03-31T15:55:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b96a00640 1 -- 172.21.15.90:0/2253332439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80003840 con 0x7f7b7c004810 2024-03-31T15:55:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 -- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 msgr2=0x7f7b7c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b7c004c90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7b80001780 tx=0x7f7b8001c400 comp rx=0 tx=0).stop 2024-03-31T15:55:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 -- 172.21.15.90:0/2253332439 shutdown_connections 2024-03-31T15:55:34.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 --2- 172.21.15.90:0/2253332439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b7c004c90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 -- 172.21.15.90:0/2253332439 >> 172.21.15.90:0/2253332439 conn(0x7f7b980b1c80 msgr2=0x7f7b980b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 -- 172.21.15.90:0/2253332439 shutdown_connections 2024-03-31T15:55:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.104+0000 7f7b9e400640 1 -- 172.21.15.90:0/2253332439 wait complete. 2024-03-31T15:55:34.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 Processor -- start 2024-03-31T15:55:34.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 -- start start 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b981543a0 0x7f7b98154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b98161080 0x7f7b98161500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9806bbb0 con 0x7f7b7c004810 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9806ba30 con 0x7f7b981543a0 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9806bd30 con 0x7f7b98161080 2024-03-31T15:55:34.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54466/0 (socket says 172.21.15.90:54466) 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 -- 172.21.15.90:0/679247126 learned_addr learned my addr 172.21.15.90:0/679247126 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b981543a0 0x7f7b98154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97e00640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b98161080 0x7f7b98161500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 -- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b98161080 msgr2=0x7f7b98161500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b98161080 0x7f7b98161500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 -- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b981543a0 msgr2=0x7f7b98154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b981543a0 0x7f7b98154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 -- 172.21.15.90:0/679247126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b98154f60 con 0x7f7b7c004810 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b97400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7b88010840 tx=0x7f7b88010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8801a020 con 0x7f7b7c004810 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8800eb60 con 0x7f7b7c004810 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800f040 con 0x7f7b7c004810 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.108+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b98155240 con 0x7f7b7c004810 2024-03-31T15:55:34.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.112+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b980cce20 con 0x7f7b7c004810 2024-03-31T15:55:34.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.112+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b980686d0 con 0x7f7b7c004810 2024-03-31T15:55:34.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.112+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b8801d020 con 0x7f7b7c004810 2024-03-31T15:55:34.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.112+0000 7f7b8f400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 0x7f7b7406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.112+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f7b880876f0 con 0x7f7b7c004810 2024-03-31T15:55:34.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.116+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 0x7f7b7406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.116+0000 7f7b8fe00640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 0x7f7b7406a840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b80001750 tx=0x7f7b800011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.116+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b88024020 con 0x7f7b7c004810 2024-03-31T15:55:34.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.276+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b980b2200 con 0x7f7b7c004810 2024-03-31T15:55:34.271 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.276+0000 7f7b8f400640 1 -- 172.21.15.90:0/679247126 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f7b88084960 con 0x7f7b7c004810 2024-03-31T15:55:34.271 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:34.271 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:34.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 msgr2=0x7f7b7406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 0x7f7b7406a840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b80001750 tx=0x7f7b800011a0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 msgr2=0x7f7b98152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7b88010840 tx=0x7f7b88010d00 comp rx=0 tx=0).stop 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 shutdown_connections 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b74068380 0x7f7b7406a840 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b98161080 0x7f7b98161500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b981543a0 0x7f7b98154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 --2- 172.21.15.90:0/679247126 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c004810 0x7f7b98152980 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 >> 172.21.15.90:0/679247126 conn(0x7f7b980b1c80 msgr2=0x7f7b980af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 shutdown_connections 2024-03-31T15:55:34.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.280+0000 7f7b9e400640 1 -- 172.21.15.90:0/679247126 wait complete. 2024-03-31T15:55:34.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:34.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 Processor -- start 2024-03-31T15:55:34.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 -- start start 2024-03-31T15:55:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8c8015e4e0 0x7f8c8006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c8006d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c8015e8c0 con 0x7f8c8006ce50 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8c800d0ff0 con 0x7f8c8015e4e0 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c86e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8c800d12d0 con 0x7f8c800cdf20 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8c8006ce50 0x7f8c800ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41704/0 (socket says 172.21.15.90:41704) 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7f400640 1 -- 172.21.15.90:0/1825531677 learned_addr learned my addr 172.21.15.90:0/1825531677 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7fe00640 1 --2- 172.21.15.90:0/1825531677 >> v2:172.21.15.102:3300/0 conn(0x7f8c8015e4e0 0x7f8c8006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c84e00640 1 --2- 172.21.15.90:0/1825531677 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c8006d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593228134 0 0) 0x7f8c8015e8c0 con 0x7f8c8006ce50 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c64003530 con 0x7f8c8006ce50 2024-03-31T15:55:34.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c84e00640 1 -- 172.21.15.90:0/1825531677 >> v2:172.21.15.102:3300/0 conn(0x7f8c8015e4e0 msgr2=0x7f8c8006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c84e00640 1 --2- 172.21.15.90:0/1825531677 >> v2:172.21.15.102:3300/0 conn(0x7f8c8015e4e0 0x7f8c8006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c84e00640 1 -- 172.21.15.90:0/1825531677 >> v1:172.21.15.90:6789/0 conn(0x7f8c8006ce50 legacy=0x7f8c800ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 657916744 0 0) 0x7f8c64003530 con 0x7f8c8006ce50 2024-03-31T15:55:34.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.440+0000 7f8c84e00640 1 -- 172.21.15.90:0/1825531677 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c800cc710 con 0x7f8c800cdf20 2024-03-31T15:55:34.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c84e00640 1 --2- 172.21.15.90:0/1825531677 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c8006d2c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009dc0 tx=0x7f8c7801c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5174aed4c132e2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780b2070 con 0x7f8c800cdf20 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c780a5890 con 0x7f8c800cdf20 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c7ea00640 1 -- 172.21.15.90:0/1825531677 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780a5c60 con 0x7f8c800cdf20 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 -- 172.21.15.90:0/1825531677 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 msgr2=0x7f8c8006d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 --2- 172.21.15.90:0/1825531677 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c8006d2c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009dc0 tx=0x7f8c7801c8c0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 -- 172.21.15.90:0/1825531677 shutdown_connections 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 --2- 172.21.15.90:0/1825531677 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c8006d2c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 --2- 172.21.15.90:0/1825531677 >> v2:172.21.15.102:3300/0 conn(0x7f8c8015e4e0 0x7f8c8006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 -- 172.21.15.90:0/1825531677 >> 172.21.15.90:0/1825531677 conn(0x7f8c800d1f20 msgr2=0x7f8c800d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 -- 172.21.15.90:0/1825531677 shutdown_connections 2024-03-31T15:55:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.444+0000 7f8c86e00640 1 -- 172.21.15.90:0/1825531677 wait complete. 2024-03-31T15:55:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 Processor -- start 2024-03-31T15:55:34.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- start start 2024-03-31T15:55:34.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 0x7f8c800cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c800c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8016b1b0 con 0x7f8c8015e4e0 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8016b030 con 0x7f8c8006ce50 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8016b330 con 0x7f8c800cdf20 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54482/0 (socket says 172.21.15.90:54482) 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c7fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 0x7f8c800cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 -- 172.21.15.90:0/999897782 learned_addr learned my addr 172.21.15.90:0/999897782 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c7f400640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c800c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 -- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 msgr2=0x7f8c800c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c800c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 -- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 msgr2=0x7f8c800cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 0x7f8c800cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 -- 172.21.15.90:0/999897782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c800c9e90 con 0x7f8c8015e4e0 2024-03-31T15:55:34.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c7fe00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 0x7f8c800cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:34.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c84e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c7801af70 tx=0x7f8c78019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780b2050 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c780be070 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c800c6d80 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780ad850 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c800c7340 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.448+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c800685f0 con 0x7f8c8015e4e0 2024-03-31T15:55:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.452+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8c780ad9f0 con 0x7f8c8015e4e0 2024-03-31T15:55:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.452+0000 7f8c77e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.452+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f8c78121b40 con 0x7f8c8015e4e0 2024-03-31T15:55:34.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.452+0000 7f8c7fe00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.452+0000 7f8c7fe00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 0x7f8c6006a840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009eb0 tx=0x7f8c70009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.460+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8c800c9e90 con 0x7f8c8015e4e0 2024-03-31T15:55:34.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8c8006f970 con 0x7f8c8015e4e0 2024-03-31T15:55:34.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c77e00640 1 -- 172.21.15.90:0/999897782 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7f8c78016540 con 0x7f8c8015e4e0 2024-03-31T15:55:34.617 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 msgr2=0x7f8c6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 0x7f8c6006a840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009eb0 tx=0x7f8c70009190 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 msgr2=0x7f8c8015b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c7801af70 tx=0x7f8c78019660 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 shutdown_connections 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c8015e4e0 0x7f8c8015b500 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8c800cdf20 0x7f8c800c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 --2- 172.21.15.90:0/999897782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c8006ce50 0x7f8c800cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 >> 172.21.15.90:0/999897782 conn(0x7f8c800d1f20 msgr2=0x7f8c800d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 shutdown_connections 2024-03-31T15:55:34.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.624+0000 7f8c86e00640 1 -- 172.21.15.90:0/999897782 wait complete. 2024-03-31T15:55:34.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 Processor -- start 2024-03-31T15:55:34.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 -- start start 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa700ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa7006d460 0x7ffa700cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7015e8c0 con 0x7ffa7015e4e0 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ffa700d0ff0 con 0x7ffa7006d460 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa75800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffa700d12d0 con 0x7ffa7006d020 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6ea00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ffa7015e4e0 0x7ffa7006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41716/0 (socket says 172.21.15.90:41716) 2024-03-31T15:55:34.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6ea00640 1 -- 172.21.15.90:0/3306101400 learned_addr learned my addr 172.21.15.90:0/3306101400 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa67e00640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa700ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6f400640 1 --2- 172.21.15.90:0/3306101400 >> v2:172.21.15.102:3300/0 conn(0x7ffa7006d460 0x7ffa700cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762560041 0 0) 0x7ffa7015e8c0 con 0x7ffa7015e4e0 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa54003530 con 0x7ffa7015e4e0 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1530551555 0 0) 0x7ffa54003530 con 0x7ffa7015e4e0 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 msgr2=0x7ffa700ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa700ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 >> v2:172.21.15.102:3300/0 conn(0x7ffa7006d460 msgr2=0x7ffa700cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 --2- 172.21.15.90:0/3306101400 >> v2:172.21.15.102:3300/0 conn(0x7ffa7006d460 0x7ffa700cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa700cc710 con 0x7ffa7015e4e0 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.784+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7ffa60001680 con 0x7ffa7015e4e0 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 >> v1:172.21.15.90:6789/0 conn(0x7ffa7015e4e0 legacy=0x7ffa7006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa540039c0 0x7ffa54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa54004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa54004d50 0x7ffa540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700d0ff0 con 0x7ffa54004d50 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700d12d0 con 0x7ffa540039c0 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5400ba00 con 0x7ffa7006d020 2024-03-31T15:55:34.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa54004d50 0x7ffa540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6ea00640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa540039c0 0x7ffa54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa67e00640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa54004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 -- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 msgr2=0x7ffa54004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa7006d020 0x7ffa54004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 -- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa540039c0 msgr2=0x7ffa54003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa540039c0 0x7ffa54003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 -- 172.21.15.90:0/3306101400 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa54003530 con 0x7ffa54004d50 2024-03-31T15:55:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6f400640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa54004d50 0x7ffa540051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009f90 tx=0x7ffa6801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54d531659e203b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa680b0070 con 0x7ffa54004d50 2024-03-31T15:55:34.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa680ab600 con 0x7ffa54004d50 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.788+0000 7ffa6e000640 1 -- 172.21.15.90:0/3306101400 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68009c10 con 0x7ffa54004d50 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- 172.21.15.90:0/3306101400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa54004d50 msgr2=0x7ffa540051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 --2- 172.21.15.90:0/3306101400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa54004d50 0x7ffa540051d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009f90 tx=0x7ffa6801c960 comp rx=0 tx=0).stop 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa6f400640 1 -- 172.21.15.90:0/3306101400 reap_dead start 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- 172.21.15.90:0/3306101400 shutdown_connections 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- 172.21.15.90:0/3306101400 >> 172.21.15.90:0/3306101400 conn(0x7ffa700d1f20 msgr2=0x7ffa700d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- 172.21.15.90:0/3306101400 shutdown_connections 2024-03-31T15:55:34.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- 172.21.15.90:0/3306101400 wait complete. 2024-03-31T15:55:34.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 Processor -- start 2024-03-31T15:55:34.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- start start 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa54004d50 0x7ffa701e9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa701480a0 0x7ffa70148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700d0d90 con 0x7ffa701591f0 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700d0c10 con 0x7ffa701480a0 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa75800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ffa700cdda0 con 0x7ffa54004d50 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54500/0 (socket says 172.21.15.90:54500) 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 -- 172.21.15.90:0/2148336650 learned_addr learned my addr 172.21.15.90:0/2148336650 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:34.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 -- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa54004d50 msgr2=0x7ffa701e9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa54004d50 0x7ffa701e9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 -- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa701480a0 msgr2=0x7ffa70148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa701480a0 0x7ffa70148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.792+0000 7ffa67e00640 1 -- 172.21.15.90:0/2148336650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa700c8ff0 con 0x7ffa701591f0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa67e00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffa58011a00 tx=0x7ffa58011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5800fa70 con 0x7ffa701591f0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5800fc10 con 0x7ffa701591f0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa58014440 con 0x7ffa701591f0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa700c92d0 con 0x7ffa701591f0 2024-03-31T15:55:34.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa700c9890 con 0x7ffa701591f0 2024-03-31T15:55:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa700685f0 con 0x7ffa701591f0 2024-03-31T15:55:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa58012040 con 0x7ffa701591f0 2024-03-31T15:55:34.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 0x7ffa4c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ffa58087380 con 0x7ffa701591f0 2024-03-31T15:55:34.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.796+0000 7ffa6ea00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 0x7ffa4c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:34.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.800+0000 7ffa6ea00640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 0x7ffa4c06a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001060 tx=0x7ffa60004810 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:34.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.800+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ffa58019030 con 0x7ffa701591f0 2024-03-31T15:55:34.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.956+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa7015e8c0 con 0x7ffa701591f0 2024-03-31T15:55:34.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa66a00640 1 -- 172.21.15.90:0/2148336650 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7ffa5801c360 con 0x7ffa701591f0 2024-03-31T15:55:34.953 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 msgr2=0x7ffa4c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 0x7ffa4c06a870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ffa60001060 tx=0x7ffa60004810 comp rx=0 tx=0).stop 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 msgr2=0x7ffa700ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ffa58011a00 tx=0x7ffa58011ec0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 shutdown_connections 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ffa4c0683b0 0x7ffa4c06a870 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa701480a0 0x7ffa70148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa701591f0 0x7ffa700ca370 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 --2- 172.21.15.90:0/2148336650 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ffa54004d50 0x7ffa701e9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 >> 172.21.15.90:0/2148336650 conn(0x7ffa700d1f20 msgr2=0x7ffa700c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 shutdown_connections 2024-03-31T15:55:34.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:34.960+0000 7ffa75800640 1 -- 172.21.15.90:0/2148336650 wait complete. 2024-03-31T15:55:34.973 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:35.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.116+0000 7f209b000640 1 Processor -- start 2024-03-31T15:55:35.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.116+0000 7f209b000640 1 -- start start 2024-03-31T15:55:35.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f209b000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f209b000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f209b000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20940cd900 con 0x7f20940cd470 2024-03-31T15:55:35.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f209b000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f20940aed70 con 0x7f209415c320 2024-03-31T15:55:35.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f209b000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f20940cefc0 con 0x7f20940d2b20 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f208be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:35174/0 (socket says 172.21.15.90:35174) 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2099000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f20940cd470 0x7f20940cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:41214/0 (socket says 172.21.15.90:41214) 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 -- 172.21.15.90:0/1287696258 learned_addr learned my addr 172.21.15.90:0/1287696258 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:35.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f208be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:46808/0 (socket says 172.21.15.90:46808) 2024-03-31T15:55:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 -- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 msgr2=0x7f20940ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 --2- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 -- 172.21.15.90:0/1287696258 >> v1:172.21.15.102:6789/0 conn(0x7f20940cd470 legacy=0x7f20940cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093e00640 1 -- 172.21.15.90:0/1287696258 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20940cf2a0 con 0x7f209415c320 2024-03-31T15:55:35.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.120+0000 7f2093400640 1 -- 172.21.15.90:0/1287696258 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543583865 0 0) 0x7f20940cd900 con 0x7f20940cd470 2024-03-31T15:55:35.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f2093e00640 1 --2- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f20940cc9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2084009870 tx=0x7f208401c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c025da84e9c64d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f2093400640 1 -- 172.21.15.90:0/1287696258 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20840b2070 con 0x7f209415c320 2024-03-31T15:55:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f2093400640 1 -- 172.21.15.90:0/1287696258 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20840ad680 con 0x7f209415c320 2024-03-31T15:55:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f2093400640 1 -- 172.21.15.90:0/1287696258 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20840ada50 con 0x7f209415c320 2024-03-31T15:55:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 -- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 msgr2=0x7f20940cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 --2- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f20940cc9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2084009870 tx=0x7f208401c4d0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 -- 172.21.15.90:0/1287696258 shutdown_connections 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 --2- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f20940ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 --2- 172.21.15.90:0/1287696258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f20940cc9f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 -- 172.21.15.90:0/1287696258 >> 172.21.15.90:0/1287696258 conn(0x7f20940d1f20 msgr2=0x7f20940d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.124+0000 7f209b000640 1 -- 172.21.15.90:0/1287696258 shutdown_connections 2024-03-31T15:55:35.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- 172.21.15.90:0/1287696258 wait complete. 2024-03-31T15:55:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 Processor -- start 2024-03-31T15:55:35.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- start start 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f2094153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f2094157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2094168ec0 con 0x7f20940cd470 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f20940d0eb0 con 0x7f209415c320 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2094169040 con 0x7f20940d2b20 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54508/0 (socket says 172.21.15.90:54508) 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 -- 172.21.15.90:0/3395920692 learned_addr learned my addr 172.21.15.90:0/3395920692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2099000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f2094157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f208be00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f2094153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 -- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 msgr2=0x7f2094153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f2094153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 -- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 msgr2=0x7f2094157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f2094157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 -- 172.21.15.90:0/3395920692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20941565c0 con 0x7f20940cd470 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2099000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f2094157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2093e00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f208401ab80 tx=0x7f208401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20840b2040 con 0x7f20940cd470 2024-03-31T15:55:35.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20840be070 con 0x7f20940cd470 2024-03-31T15:55:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20941568a0 con 0x7f20940cd470 2024-03-31T15:55:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.128+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20840b8420 con 0x7f20940cd470 2024-03-31T15:55:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2094156e60 con 0x7f20940cd470 2024-03-31T15:55:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20940685f0 con 0x7f20940cd470 2024-03-31T15:55:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f20840b85c0 con 0x7f20940cd470 2024-03-31T15:55:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f2090c00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f2084121100 con 0x7f20940cd470 2024-03-31T15:55:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f208be00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.132+0000 7f208be00640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f209415f860 tx=0x7f207c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.136+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2084016150 con 0x7f20940cd470 2024-03-31T15:55:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f20941584d0 con 0x7f20940cd470 2024-03-31T15:55:35.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f2090c00640 1 -- 172.21.15.90:0/3395920692 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f20840f11e0 con 0x7f20940cd470 2024-03-31T15:55:35.289 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 msgr2=0x7f207006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f209415f860 tx=0x7f207c008040 comp rx=0 tx=0).stop 2024-03-31T15:55:35.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 msgr2=0x7f2094166940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f208401ab80 tx=0x7f208401ad10 comp rx=0 tx=0).stop 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 shutdown_connections 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2070068380 0x7f207006a840 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f209415c320 0x7f2094157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f20940d2b20 0x7f2094153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 --2- 172.21.15.90:0/3395920692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f20940cd470 0x7f2094166940 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 >> 172.21.15.90:0/3395920692 conn(0x7f20940d1f20 msgr2=0x7f20940cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 shutdown_connections 2024-03-31T15:55:35.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.296+0000 7f209b000640 1 -- 172.21.15.90:0/3395920692 wait complete. 2024-03-31T15:55:35.310 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:55:35.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 Processor -- start 2024-03-31T15:55:35.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 -- start start 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45b806d020 0x7f45b80ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f45b806d490 0x7f45b80cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45b815e8c0 con 0x7f45b815e4e0 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f45b80d0ff0 con 0x7f45b806d490 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45bd200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f45b80d12d0 con 0x7f45b806d020 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b5e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f45b815e4e0 0x7f45b806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41718/0 (socket says 172.21.15.90:41718) 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b5e00640 1 -- 172.21.15.90:0/2323416217 learned_addr learned my addr 172.21.15.90:0/2323416217 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b6800640 1 --2- 172.21.15.90:0/2323416217 >> v2:172.21.15.90:3300/0 conn(0x7f45b806d490 0x7f45b80cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b5400640 1 --2- 172.21.15.90:0/2323416217 >> v2:172.21.15.102:3300/0 conn(0x7f45b806d020 0x7f45b80ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b6800640 1 -- 172.21.15.90:0/2323416217 >> v2:172.21.15.102:3300/0 conn(0x7f45b806d020 msgr2=0x7f45b80ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:35.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b6800640 1 --2- 172.21.15.90:0/2323416217 >> v2:172.21.15.102:3300/0 conn(0x7f45b806d020 0x7f45b80ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b6800640 1 -- 172.21.15.90:0/2323416217 >> v1:172.21.15.90:6789/0 conn(0x7f45b815e4e0 legacy=0x7f45b806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.467 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.472+0000 7f45b6800640 1 -- 172.21.15.90:0/2323416217 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b80cc710 con 0x7f45b806d490 2024-03-31T15:55:35.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45b6800640 1 --2- 172.21.15.90:0/2323416217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d490 0x7f45b80cd450 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009d90 tx=0x7f45b001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d777143c37cf9f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45b4a00640 1 -- 172.21.15.90:0/2323416217 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b00b2070 con 0x7f45b806d490 2024-03-31T15:55:35.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45b4a00640 1 -- 172.21.15.90:0/2323416217 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b00a5760 con 0x7f45b806d490 2024-03-31T15:55:35.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45b4a00640 1 -- 172.21.15.90:0/2323416217 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b00a5b30 con 0x7f45b806d490 2024-03-31T15:55:35.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- 172.21.15.90:0/2323416217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d490 msgr2=0x7f45b80cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 --2- 172.21.15.90:0/2323416217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d490 0x7f45b80cd450 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009d90 tx=0x7f45b001c8a0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- 172.21.15.90:0/2323416217 shutdown_connections 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 --2- 172.21.15.90:0/2323416217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d490 0x7f45b80cd450 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 --2- 172.21.15.90:0/2323416217 >> v2:172.21.15.102:3300/0 conn(0x7f45b806d020 0x7f45b80ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- 172.21.15.90:0/2323416217 >> 172.21.15.90:0/2323416217 conn(0x7f45b80d1f20 msgr2=0x7f45b80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- 172.21.15.90:0/2323416217 shutdown_connections 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- 172.21.15.90:0/2323416217 wait complete. 2024-03-31T15:55:35.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 Processor -- start 2024-03-31T15:55:35.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.476+0000 7f45bd200640 1 -- start start 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45b806d490 0x7f45b80cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 0x7f45b806f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f45b80d0d90 con 0x7f45b806d020 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45b80d0c10 con 0x7f45b806d490 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f45b816b030 con 0x7f45b815e4e0 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54516/0 (socket says 172.21.15.90:54516) 2024-03-31T15:55:35.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 -- 172.21.15.90:0/4139901142 learned_addr learned my addr 172.21.15.90:0/4139901142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5400640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45b806d490 0x7f45b80cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b6800640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 0x7f45b806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 -- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 msgr2=0x7f45b806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 0x7f45b806f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 -- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45b806d490 msgr2=0x7f45b80cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45b806d490 0x7f45b80cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 -- 172.21.15.90:0/4139901142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b8153470 con 0x7f45b806d020 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b6800640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 0x7f45b806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45b5e00640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c450 tx=0x7f45a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8017070 con 0x7f45b806d020 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a8005cf0 con 0x7f45b806d020 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b8158220 con 0x7f45b806d020 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8007290 con 0x7f45b806d020 2024-03-31T15:55:35.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b81587e0 con 0x7f45b806d020 2024-03-31T15:55:35.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b80685f0 con 0x7f45b806d020 2024-03-31T15:55:35.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.480+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f45a801a070 con 0x7f45b806d020 2024-03-31T15:55:35.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.484+0000 7f45ae000640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 0x7f459806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.484+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f45a8086090 con 0x7f45b806d020 2024-03-31T15:55:35.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.484+0000 7f45b5400640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 0x7f459806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.484+0000 7f45b5400640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 0x7f459806a840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45b8156a50 tx=0x7f45a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.488+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f45b8153470 con 0x7f45b806d020 2024-03-31T15:55:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.644+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f45b80d2910 con 0x7f45b806d020 2024-03-31T15:55:35.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.644+0000 7f45ae000640 1 -- 172.21.15.90:0/4139901142 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f45a8056280 con 0x7f45b806d020 2024-03-31T15:55:35.640 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 msgr2=0x7f459806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 0x7f459806a840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45b8156a50 tx=0x7f45a4008040 comp rx=0 tx=0).stop 2024-03-31T15:55:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 msgr2=0x7f45b8152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c450 tx=0x7f45a800c910 comp rx=0 tx=0).stop 2024-03-31T15:55:35.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 shutdown_connections 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4598068380 0x7f459806a840 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f45b815e4e0 0x7f45b806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45b806d490 0x7f45b80cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 --2- 172.21.15.90:0/4139901142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f45b806d020 0x7f45b8152f30 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 >> 172.21.15.90:0/4139901142 conn(0x7f45b80d1f20 msgr2=0x7f45b80c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 shutdown_connections 2024-03-31T15:55:35.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.648+0000 7f45bd200640 1 -- 172.21.15.90:0/4139901142 wait complete. 2024-03-31T15:55:35.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:55:35.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 Processor -- start 2024-03-31T15:55:35.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 -- start start 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe8640ce910 0x7fe8640cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe86415d240 0x7fe8640ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8640cf230 con 0x7fe8640cfe90 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe8640cd460 con 0x7fe8640ce910 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86c600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe8640cd740 con 0x7fe86415d240 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe8640ce910 0x7fe8640cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86a600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe8640cfe90 0x7fe8640d02c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41722/0 (socket says 172.21.15.90:41722) 2024-03-31T15:55:35.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe86a600640 1 -- 172.21.15.90:0/2607735407 learned_addr learned my addr 172.21.15.90:0/2607735407 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869200640 1 --2- 172.21.15.90:0/2607735407 >> v2:172.21.15.102:3300/0 conn(0x7fe86415d240 0x7fe8640ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869c00640 1 -- 172.21.15.90:0/2607735407 >> v2:172.21.15.102:3300/0 conn(0x7fe86415d240 msgr2=0x7fe8640ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869c00640 1 --2- 172.21.15.90:0/2607735407 >> v2:172.21.15.102:3300/0 conn(0x7fe86415d240 0x7fe8640ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869c00640 1 -- 172.21.15.90:0/2607735407 >> v1:172.21.15.90:6789/0 conn(0x7fe8640cfe90 legacy=0x7fe8640d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe869c00640 1 -- 172.21.15.90:0/2607735407 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8640cda20 con 0x7fe8640ce910 2024-03-31T15:55:35.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.808+0000 7fe863e00640 1 -- 172.21.15.90:0/2607735407 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2441689296 0 0) 0x7fe8640cf230 con 0x7fe8640cfe90 2024-03-31T15:55:35.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe869c00640 1 --2- 172.21.15.90:0/2607735407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe8640cecf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe864072f50 tx=0x7fe85801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d477b306aded1c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe863e00640 1 -- 172.21.15.90:0/2607735407 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580b0070 con 0x7fe8640ce910 2024-03-31T15:55:35.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe863e00640 1 -- 172.21.15.90:0/2607735407 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8580ab600 con 0x7fe8640ce910 2024-03-31T15:55:35.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe863e00640 1 -- 172.21.15.90:0/2607735407 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580ab9d0 con 0x7fe8640ce910 2024-03-31T15:55:35.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 -- 172.21.15.90:0/2607735407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 msgr2=0x7fe8640cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 --2- 172.21.15.90:0/2607735407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe8640cecf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe864072f50 tx=0x7fe85801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 -- 172.21.15.90:0/2607735407 shutdown_connections 2024-03-31T15:55:35.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 --2- 172.21.15.90:0/2607735407 >> v2:172.21.15.102:3300/0 conn(0x7fe86415d240 0x7fe8640ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 --2- 172.21.15.90:0/2607735407 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe8640cecf0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 -- 172.21.15.90:0/2607735407 >> 172.21.15.90:0/2607735407 conn(0x7fe86408bfd0 msgr2=0x7fe8640d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 -- 172.21.15.90:0/2607735407 shutdown_connections 2024-03-31T15:55:35.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.812+0000 7fe86c600640 1 -- 172.21.15.90:0/2607735407 wait complete. 2024-03-31T15:55:35.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 Processor -- start 2024-03-31T15:55:35.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 -- start start 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8640cfe90 0x7fe8641f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe86415d240 0x7fe8641f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe8640cd1d0 con 0x7fe8640ce910 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe8640cd050 con 0x7fe8640cfe90 2024-03-31T15:55:35.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86c600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe864169410 con 0x7fe86415d240 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54534/0 (socket says 172.21.15.90:54534) 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 -- 172.21.15.90:0/1872603967 learned_addr learned my addr 172.21.15.90:0/1872603967 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869200640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8640cfe90 0x7fe8641f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe86a600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe86415d240 0x7fe8641f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 -- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe86415d240 msgr2=0x7fe8641f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe86415d240 0x7fe8641f8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 -- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8640cfe90 msgr2=0x7fe8641f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8640cfe90 0x7fe8641f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 -- 172.21.15.90:0/1872603967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8640c7140 con 0x7fe8640ce910 2024-03-31T15:55:35.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe869c00640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe858019ff0 tx=0x7fe85801ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.816+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580b0050 con 0x7fe8640ce910 2024-03-31T15:55:35.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8580bc070 con 0x7fe8640ce910 2024-03-31T15:55:35.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8640c7420 con 0x7fe8640ce910 2024-03-31T15:55:35.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858006e20 con 0x7fe8640ce910 2024-03-31T15:55:35.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8640b4e00 con 0x7fe8640ce910 2024-03-31T15:55:35.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe864068660 con 0x7fe8640ce910 2024-03-31T15:55:35.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe8580b5830 con 0x7fe8640ce910 2024-03-31T15:55:35.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe861600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 0x7fe84406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:35.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fe858121430 con 0x7fe8640ce910 2024-03-31T15:55:35.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.820+0000 7fe869200640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 0x7fe84406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:35.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.824+0000 7fe869200640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 0x7fe84406a840 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe8640b7020 tx=0x7fe850009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:35.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.824+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe858016070 con 0x7fe8640ce910 2024-03-31T15:55:35.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.992+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe8640ce6f0 con 0x7fe8640ce910 2024-03-31T15:55:35.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe861600640 1 -- 172.21.15.90:0/1872603967 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7fe8580f1060 con 0x7fe8640ce910 2024-03-31T15:55:35.988 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:55:35.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 msgr2=0x7fe84406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 0x7fe84406a840 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe8640b7020 tx=0x7fe850009190 comp rx=0 tx=0).stop 2024-03-31T15:55:35.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 msgr2=0x7fe86414b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:35.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe858019ff0 tx=0x7fe85801ac00 comp rx=0 tx=0).stop 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 shutdown_connections 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe844068380 0x7fe84406a840 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe86415d240 0x7fe8641f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8640cfe90 0x7fe8641f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 --2- 172.21.15.90:0/1872603967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8640ce910 0x7fe86414b310 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 >> 172.21.15.90:0/1872603967 conn(0x7fe86408bfd0 msgr2=0x7fe86406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 shutdown_connections 2024-03-31T15:55:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:35.996+0000 7fe86c600640 1 -- 172.21.15.90:0/1872603967 wait complete. 2024-03-31T15:55:36.018 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:55:36.018 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:55:36.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:36.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 Processor -- start 2024-03-31T15:55:36.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 -- start start 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc24c15e1b0 0x7fc24c162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c06c7b0 0x7fc24c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24c15f780 con 0x7fc24c06d5f0 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc24c06daa0 con 0x7fc24c15e1b0 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.212+0000 7fc253c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c1622b0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc24c06d5f0 0x7fc24c15e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41738/0 (socket says 172.21.15.90:41738) 2024-03-31T15:55:36.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251c00640 1 -- 172.21.15.90:0/162575995 learned_addr learned my addr 172.21.15.90:0/162575995 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:36.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 --2- 172.21.15.90:0/162575995 >> v2:172.21.15.102:3300/0 conn(0x7fc24c15e1b0 0x7fc24c162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc24be00640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c06c7b0 0x7fc24c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081566118 0 0) 0x7fc24c15f780 con 0x7fc24c06d5f0 2024-03-31T15:55:36.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc234003530 con 0x7fc24c06d5f0 2024-03-31T15:55:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3044331693 0 0) 0x7fc234003530 con 0x7fc24c06d5f0 2024-03-31T15:55:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 -- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c06c7b0 msgr2=0x7fc24c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c06c7b0 0x7fc24c06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 -- 172.21.15.90:0/162575995 >> v1:172.21.15.90:6789/0 conn(0x7fc24c06d5f0 legacy=0x7fc24c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 -- 172.21.15.90:0/162575995 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c15f1c0 con 0x7fc24c15e1b0 2024-03-31T15:55:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.216+0000 7fc251200640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c15e1b0 0x7fc24c162740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc2400097c0 tx=0x7fc24001c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a95e283423ccf60 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400b0070 con 0x7fc24c15e1b0 2024-03-31T15:55:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2400ab400 con 0x7fc24c15e1b0 2024-03-31T15:55:36.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc24b400640 1 -- 172.21.15.90:0/162575995 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244001a00 con 0x7fc24c15e1b0 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c15e1b0 msgr2=0x7fc24c162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c15e1b0 0x7fc24c162740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc2400097c0 tx=0x7fc24001c300 comp rx=0 tx=0).stop 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- 172.21.15.90:0/162575995 shutdown_connections 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c06c7b0 0x7fc24c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 --2- 172.21.15.90:0/162575995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c15e1b0 0x7fc24c162740 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- 172.21.15.90:0/162575995 >> 172.21.15.90:0/162575995 conn(0x7fc24c09aa40 msgr2=0x7fc24c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:36.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- 172.21.15.90:0/162575995 shutdown_connections 2024-03-31T15:55:36.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- 172.21.15.90:0/162575995 wait complete. 2024-03-31T15:55:36.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 Processor -- start 2024-03-31T15:55:36.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.220+0000 7fc253c00640 1 -- start start 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c06d5f0 0x7fc24c0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c15e1b0 0x7fc24c0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c0d2940 con 0x7fc24c06c7b0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c0d27c0 con 0x7fc24c06d5f0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c0d2ac0 con 0x7fc24c15e1b0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54538/0 (socket says 172.21.15.90:54538) 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 -- 172.21.15.90:0/607104824 learned_addr learned my addr 172.21.15.90:0/607104824 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc24be00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c06d5f0 0x7fc24c0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c15e1b0 0x7fc24c0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 -- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c15e1b0 msgr2=0x7fc24c0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c15e1b0 0x7fc24c0cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 -- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c06d5f0 msgr2=0x7fc24c0cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c06d5f0 0x7fc24c0cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 -- 172.21.15.90:0/607104824 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c0af360 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc251200640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc24001a9b0 tx=0x7fc2400190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400ba070 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2400038a0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c0af640 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2400b0040 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c0af050 con 0x7fc24c06c7b0 2024-03-31T15:55:36.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.224+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c0686d0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.228+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc2400bd070 con 0x7fc24c06c7b0 2024-03-31T15:55:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.228+0000 7fc248c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.228+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fc24011ffd0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.228+0000 7fc24be00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.228+0000 7fc24be00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 0x7fc22c06a840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0ce220 tx=0x7fc238008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.232+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc240015fb0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.392+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc24c0af810 con 0x7fc24c06c7b0 2024-03-31T15:55:36.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.392+0000 7fc248c00640 1 -- 172.21.15.90:0/607104824 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7fc2400f01c0 con 0x7fc24c06c7b0 2024-03-31T15:55:36.386 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:36.387 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:36.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 msgr2=0x7fc22c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 0x7fc22c06a840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0ce220 tx=0x7fc238008040 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 msgr2=0x7fc24c0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc24001a9b0 tx=0x7fc2400190a0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 shutdown_connections 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc22c068380 0x7fc22c06a840 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc24c15e1b0 0x7fc24c0cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc24c06d5f0 0x7fc24c0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 --2- 172.21.15.90:0/607104824 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc24c06c7b0 0x7fc24c0aeb10 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 >> 172.21.15.90:0/607104824 conn(0x7fc24c09aa40 msgr2=0x7fc24c0d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 shutdown_connections 2024-03-31T15:55:36.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.396+0000 7fc253c00640 1 -- 172.21.15.90:0/607104824 wait complete. 2024-03-31T15:55:36.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:36.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:36.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 Processor -- start 2024-03-31T15:55:36.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 -- start start 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec5415aef0 0x7fec5415b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec540b2e30 0x7fec540b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec54070910 con 0x7fec5415f5c0 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fec540b37e0 con 0x7fec5415aef0 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec5a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fec54070bf0 con 0x7fec540b2e30 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec5415aef0 0x7fec5415b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fec5415aef0 0x7fec5415b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54540/0 (socket says 172.21.15.90:54540) 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 -- 172.21.15.90:0/1813082312 learned_addr learned my addr 172.21.15.90:0/1813082312 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53e00640 1 --2- 172.21.15.90:0/1813082312 >> v2:172.21.15.102:3300/0 conn(0x7fec540b2e30 0x7fec540b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 -- 172.21.15.90:0/1813082312 >> v2:172.21.15.102:3300/0 conn(0x7fec540b2e30 msgr2=0x7fec540b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 --2- 172.21.15.90:0/1813082312 >> v2:172.21.15.102:3300/0 conn(0x7fec540b2e30 0x7fec540b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 -- 172.21.15.90:0/1813082312 >> v1:172.21.15.102:6789/0 conn(0x7fec5415f5c0 legacy=0x7fec5415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.556+0000 7fec53400640 1 -- 172.21.15.90:0/1813082312 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5415b9d0 con 0x7fec5415aef0 2024-03-31T15:55:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec53400640 1 --2- 172.21.15.90:0/1813082312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec5415aef0 0x7fec5415b2d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fec44009870 tx=0x7fec4401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e565b71cdc1e412d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec52000640 1 -- 172.21.15.90:0/1813082312 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440ab5f0 con 0x7fec5415aef0 2024-03-31T15:55:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec52000640 1 -- 172.21.15.90:0/1813082312 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec440b0070 con 0x7fec5415aef0 2024-03-31T15:55:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec52000640 1 -- 172.21.15.90:0/1813082312 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440ab9b0 con 0x7fec5415aef0 2024-03-31T15:55:36.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 -- 172.21.15.90:0/1813082312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec5415aef0 msgr2=0x7fec5415b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 --2- 172.21.15.90:0/1813082312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec5415aef0 0x7fec5415b2d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fec44009870 tx=0x7fec4401c4e0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 -- 172.21.15.90:0/1813082312 shutdown_connections 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 --2- 172.21.15.90:0/1813082312 >> v2:172.21.15.102:3300/0 conn(0x7fec540b2e30 0x7fec540b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 --2- 172.21.15.90:0/1813082312 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec5415aef0 0x7fec5415b2d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 -- 172.21.15.90:0/1813082312 >> 172.21.15.90:0/1813082312 conn(0x7fec540cde90 msgr2=0x7fec540ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 -- 172.21.15.90:0/1813082312 shutdown_connections 2024-03-31T15:55:36.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 -- 172.21.15.90:0/1813082312 wait complete. 2024-03-31T15:55:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.560+0000 7fec5a800640 1 Processor -- start 2024-03-31T15:55:36.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- start start 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec5415aef0 0x7fec540cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec5415f5c0 0x7fec5414d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec540d0a10 con 0x7fec540b2e30 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec540d0890 con 0x7fec5415f5c0 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fec540af430 con 0x7fec5415aef0 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54556/0 (socket says 172.21.15.90:54556) 2024-03-31T15:55:36.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 -- 172.21.15.90:0/2605364139 learned_addr learned my addr 172.21.15.90:0/2605364139 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec52a00640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec5415aef0 0x7fec540cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53e00640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec5415f5c0 0x7fec5414d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 -- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec5415aef0 msgr2=0x7fec540cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec5415aef0 0x7fec540cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 -- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec5415f5c0 msgr2=0x7fec5414d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec5415f5c0 0x7fec5414d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 -- 172.21.15.90:0/2605364139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec540ccc40 con 0x7fec540b2e30 2024-03-31T15:55:36.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec53400640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad040 tx=0x7fec4401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440abe60 con 0x7fec540b2e30 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec440be610 con 0x7fec540b2e30 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440bea00 con 0x7fec540b2e30 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec540cbd30 con 0x7fec540b2e30 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec540cc2f0 con 0x7fec540b2e30 2024-03-31T15:55:36.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.564+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec540685f0 con 0x7fec540b2e30 2024-03-31T15:55:36.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.568+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fec440b43f0 con 0x7fec540b2e30 2024-03-31T15:55:36.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.568+0000 7fec4b400640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 0x7fec3406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.568+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fec44121080 con 0x7fec540b2e30 2024-03-31T15:55:36.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.568+0000 7fec52a00640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 0x7fec3406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.568+0000 7fec52a00640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 0x7fec3406a870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fec54156730 tx=0x7fec40009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.572+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fec440c4060 con 0x7fec540b2e30 2024-03-31T15:55:36.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec540cf760 con 0x7fec540b2e30 2024-03-31T15:55:36.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec4b400640 1 -- 172.21.15.90:0/2605364139 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7fec440160d0 con 0x7fec540b2e30 2024-03-31T15:55:36.726 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 msgr2=0x7fec3406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 0x7fec3406a870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fec54156730 tx=0x7fec40009060 comp rx=0 tx=0).stop 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 msgr2=0x7fec5414cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad040 tx=0x7fec4401ad20 comp rx=0 tx=0).stop 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 shutdown_connections 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fec340683b0 0x7fec3406a870 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec5415f5c0 0x7fec5414d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fec5415aef0 0x7fec540cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 --2- 172.21.15.90:0/2605364139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec540b2e30 0x7fec5414cbe0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 >> 172.21.15.90:0/2605364139 conn(0x7fec540cde90 msgr2=0x7fec5406b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:36.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 shutdown_connections 2024-03-31T15:55:36.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.732+0000 7fec5a800640 1 -- 172.21.15.90:0/2605364139 wait complete. 2024-03-31T15:55:36.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:36.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 Processor -- start 2024-03-31T15:55:36.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 -- start start 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70700b0210 0x7f7070166eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70700cefc0 con 0x7f707015c320 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70700cf2a0 con 0x7f70700d2b20 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.900+0000 7f7077e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f70700aed70 con 0x7f70700b0210 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f707015c320 0x7f70700cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41742/0 (socket says 172.21.15.90:41742) 2024-03-31T15:55:36.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075400640 1 -- 172.21.15.90:0/1308845251 learned_addr learned my addr 172.21.15.90:0/1308845251 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951355929 0 0) 0x7f70700cefc0 con 0x7f707015c320 2024-03-31T15:55:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7058003530 con 0x7f707015c320 2024-03-31T15:55:36.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 --2- 172.21.15.90:0/1308845251 >> v2:172.21.15.102:3300/0 conn(0x7f70700b0210 0x7f7070166eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 395451158 0 0) 0x7f7058003530 con 0x7f707015c320 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 >> v2:172.21.15.102:3300/0 conn(0x7f70700b0210 msgr2=0x7f7070166eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 --2- 172.21.15.90:0/1308845251 >> v2:172.21.15.102:3300/0 conn(0x7f70700b0210 0x7f7070166eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 >> v1:172.21.15.102:6789/0 conn(0x7f70700d2b20 legacy=0x7f70700cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70700cf580 con 0x7f707015c320 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7064001680 con 0x7f707015c320 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 >> v1:172.21.15.90:6789/0 conn(0x7f707015c320 legacy=0x7f70700cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70580039c0 0x7f7058003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7058004810 0x7f7058004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f705800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70700aed70 con 0x7f70580052f0 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f705800c1c0 con 0x7f70580039c0 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f705800c3c0 con 0x7f7058004810 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f705800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075400640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70580039c0 0x7f7058003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7074a00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7058004810 0x7f7058004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 -- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7058004810 msgr2=0x7f7058004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7058004810 0x7f7058004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 -- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70580039c0 msgr2=0x7f7058003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70580039c0 0x7f7058003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7075e00640 1 -- 172.21.15.90:0/1308845251 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7058003530 con 0x7f70580052f0 2024-03-31T15:55:36.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.904+0000 7f7074a00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7058004810 0x7f7058004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7075e00640 1 -- 172.21.15.90:0/1308845251 reap_dead start 2024-03-31T15:55:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7075e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f705800b9d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f70700b06a0 tx=0x7f706801ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7aacbd1fdbabb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068009be0 con 0x7f70580052f0 2024-03-31T15:55:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70680a5c10 con 0x7f70580052f0 2024-03-31T15:55:36.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f706fe00640 1 -- 172.21.15.90:0/1308845251 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70680ad440 con 0x7f70580052f0 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 msgr2=0x7f705800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f705800b9d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f70700b06a0 tx=0x7f706801ce20 comp rx=0 tx=0).stop 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- 172.21.15.90:0/1308845251 shutdown_connections 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 --2- 172.21.15.90:0/1308845251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f705800b9d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- 172.21.15.90:0/1308845251 >> 172.21.15.90:0/1308845251 conn(0x7f70700d1f20 msgr2=0x7f70700d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- 172.21.15.90:0/1308845251 shutdown_connections 2024-03-31T15:55:36.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- 172.21.15.90:0/1308845251 wait complete. 2024-03-31T15:55:36.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 Processor -- start 2024-03-31T15:55:36.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.908+0000 7f7077e00640 1 -- start start 2024-03-31T15:55:36.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 0x7f70701f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70701f79e0 0x7f70701f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70700af380 con 0x7f70580052f0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70700af200 con 0x7f7070149cb0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f70700af500 con 0x7f70701f79e0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54578/0 (socket says 172.21.15.90:54578) 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 -- 172.21.15.90:0/110943744 learned_addr learned my addr 172.21.15.90:0/110943744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70701f79e0 0x7f70701f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7074a00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 0x7f70701f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 -- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70701f79e0 msgr2=0x7f70701f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70701f79e0 0x7f70701f7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 -- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 msgr2=0x7f70701f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:36.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 0x7f70701f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 -- 172.21.15.90:0/110943744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70701575a0 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7074a00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 0x7f70701f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7075400640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7064012290 tx=0x7f7064012750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706401c070 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7064014770 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70700cb710 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064014b60 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70700cbcd0 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.912+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70700685f0 con 0x7f70580052f0 2024-03-31T15:55:36.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.916+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f706401f070 con 0x7f70580052f0 2024-03-31T15:55:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.916+0000 7f706d600640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 0x7f705006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.916+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f7064089590 con 0x7f70580052f0 2024-03-31T15:55:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.916+0000 7f7074a00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 0x7f705006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:36.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.916+0000 7f7074a00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 0x7f705006a840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c10 tx=0x7f705c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:36.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:36.920+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7064059130 con 0x7f70580052f0 2024-03-31T15:55:37.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70700d0610 con 0x7f70580052f0 2024-03-31T15:55:37.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f706d600640 1 -- 172.21.15.90:0/110943744 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f706405d430 con 0x7f70580052f0 2024-03-31T15:55:37.070 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:37.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 msgr2=0x7f705006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 0x7f705006a840 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c10 tx=0x7f705c009040 comp rx=0 tx=0).stop 2024-03-31T15:55:37.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 msgr2=0x7f70700cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7064012290 tx=0x7f7064012750 comp rx=0 tx=0).stop 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 shutdown_connections 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7050068380 0x7f705006a840 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f70701f79e0 0x7f70701f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7070149cb0 0x7f70701f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 --2- 172.21.15.90:0/110943744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70580052f0 0x7f70700cb130 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 >> 172.21.15.90:0/110943744 conn(0x7f70700d1f20 msgr2=0x7f70700cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.076+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 shutdown_connections 2024-03-31T15:55:37.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.080+0000 7f7077e00640 1 -- 172.21.15.90:0/110943744 wait complete. 2024-03-31T15:55:37.092 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:37.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 Processor -- start 2024-03-31T15:55:37.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 -- start start 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fce0006d020 0x7fce000ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce0015e8c0 con 0x7fce0006d490 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce000d0ff0 con 0x7fce0015e4e0 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.236+0000 7fce07000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fce000d12d0 con 0x7fce0006d020 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fce05000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fce0006d490 0x7fce000cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41748/0 (socket says 172.21.15.90:41748) 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fce05000640 1 -- 172.21.15.90:0/1561022529 learned_addr learned my addr 172.21.15.90:0/1561022529 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:37.232 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdff400640 1 --2- 172.21.15.90:0/1561022529 >> v2:172.21.15.102:3300/0 conn(0x7fce0006d020 0x7fce000ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdfea00640 1 -- 172.21.15.90:0/1561022529 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799479603 0 0) 0x7fce0015e8c0 con 0x7fce0006d490 2024-03-31T15:55:37.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdfea00640 1 -- 172.21.15.90:0/1561022529 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde4003530 con 0x7fce0006d490 2024-03-31T15:55:37.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdff400640 1 -- 172.21.15.90:0/1561022529 >> v1:172.21.15.102:6789/0 conn(0x7fce0015e4e0 legacy=0x7fce0006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdff400640 1 -- 172.21.15.90:0/1561022529 >> v1:172.21.15.90:6789/0 conn(0x7fce0006d490 legacy=0x7fce000cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.240+0000 7fcdff400640 1 -- 172.21.15.90:0/1561022529 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce000cc710 con 0x7fce0006d020 2024-03-31T15:55:37.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fcdff400640 1 --2- 172.21.15.90:0/1561022529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce000ccf10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0097f0 tx=0x7fcdec01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26e6f26e6cd14aa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fcdfea00640 1 -- 172.21.15.90:0/1561022529 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec0b0070 con 0x7fce0006d020 2024-03-31T15:55:37.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fcdfea00640 1 -- 172.21.15.90:0/1561022529 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec0ab600 con 0x7fce0006d020 2024-03-31T15:55:37.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fcdfea00640 1 -- 172.21.15.90:0/1561022529 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec0ab9d0 con 0x7fce0006d020 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 -- 172.21.15.90:0/1561022529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 msgr2=0x7fce000ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 --2- 172.21.15.90:0/1561022529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce000ccf10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0097f0 tx=0x7fcdec01c410 comp rx=0 tx=0).stop 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 -- 172.21.15.90:0/1561022529 shutdown_connections 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 --2- 172.21.15.90:0/1561022529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce000ccf10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 -- 172.21.15.90:0/1561022529 >> 172.21.15.90:0/1561022529 conn(0x7fce000d1f20 msgr2=0x7fce000d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 -- 172.21.15.90:0/1561022529 shutdown_connections 2024-03-31T15:55:37.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.244+0000 7fce07000640 1 -- 172.21.15.90:0/1561022529 wait complete. 2024-03-31T15:55:37.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 Processor -- start 2024-03-31T15:55:37.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- start start 2024-03-31T15:55:37.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce0015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 0x7fce00145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fce000d0d90 con 0x7fce0006d490 2024-03-31T15:55:37.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fce000d0c10 con 0x7fce0006d020 2024-03-31T15:55:37.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fce0016c040 con 0x7fce0015e4e0 2024-03-31T15:55:37.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54580/0 (socket says 172.21.15.90:54580) 2024-03-31T15:55:37.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 -- 172.21.15.90:0/3055219364 learned_addr learned my addr 172.21.15.90:0/3055219364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:37.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdffe00640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce0015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce05000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 0x7fce00145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 -- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 msgr2=0x7fce00145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 0x7fce00145050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 -- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 msgr2=0x7fce0015a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce0015a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 -- 172.21.15.90:0/3055219364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0015bb70 con 0x7fce0006d490 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce05000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 0x7fce00145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:37.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdff400640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcdec019020 tx=0x7fce000ce370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec0b0040 con 0x7fce0006d490 2024-03-31T15:55:37.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec0b9070 con 0x7fce0006d490 2024-03-31T15:55:37.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec0b4670 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce0015aad0 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce0015b090 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.248+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce000685f0 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.252+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcdec0b4810 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.252+0000 7fcdf7e00640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 0x7fcde006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.252+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fcdec120850 con 0x7fce0006d490 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.252+0000 7fcdffe00640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 0x7fcde006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.252+0000 7fcdffe00640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 0x7fcde006a840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fce00155da0 tx=0x7fcdf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.256+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcdec016090 con 0x7fce0006d490 2024-03-31T15:55:37.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fce000d1790 con 0x7fce0006d490 2024-03-31T15:55:37.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fcdf7e00640 1 -- 172.21.15.90:0/3055219364 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7fcdec0f0ae0 con 0x7fce0006d490 2024-03-31T15:55:37.409 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:37.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 msgr2=0x7fcde006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 0x7fcde006a840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fce00155da0 tx=0x7fcdf0009040 comp rx=0 tx=0).stop 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 msgr2=0x7fce000cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcdec019020 tx=0x7fce000ce370 comp rx=0 tx=0).stop 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 shutdown_connections 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcde0068380 0x7fcde006a840 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fce0015e4e0 0x7fce00145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fce0006d490 0x7fce000cb840 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 --2- 172.21.15.90:0/3055219364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce0006d020 0x7fce0015a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 >> 172.21.15.90:0/3055219364 conn(0x7fce000d1f20 msgr2=0x7fce000c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 shutdown_connections 2024-03-31T15:55:37.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.416+0000 7fce07000640 1 -- 172.21.15.90:0/3055219364 wait complete. 2024-03-31T15:55:37.430 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:55:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 Processor -- start 2024-03-31T15:55:37.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 -- start start 2024-03-31T15:55:37.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f667815e4e0 0x7f667806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667815e8c0 con 0x7f66780cdf20 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66780d0ff0 con 0x7f667806ce50 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f667e000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f66780d12d0 con 0x7f667815e4e0 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f667815e4e0 0x7f667806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f66780cdf20 0x7f66780ceeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41764/0 (socket says 172.21.15.90:41764) 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677e00640 1 -- 172.21.15.90:0/1049055967 learned_addr learned my addr 172.21.15.90:0/1049055967 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677400640 1 -- 172.21.15.90:0/1049055967 >> v1:172.21.15.102:6789/0 conn(0x7f667806ce50 legacy=0x7f66780ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677400640 1 -- 172.21.15.90:0/1049055967 >> v1:172.21.15.90:6789/0 conn(0x7f66780cdf20 legacy=0x7f66780ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6677400640 1 -- 172.21.15.90:0/1049055967 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66780cc710 con 0x7f667815e4e0 2024-03-31T15:55:37.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.576+0000 7f6676000640 1 -- 172.21.15.90:0/1049055967 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2106800217 0 0) 0x7f667815e8c0 con 0x7f66780cdf20 2024-03-31T15:55:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.580+0000 7f6677400640 1 --2- 172.21.15.90:0/1049055967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667815e4e0 0x7f667806c910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f66680097f0 tx=0x7f666801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903dbc4a4934ec89 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6676000640 1 -- 172.21.15.90:0/1049055967 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66680b0070 con 0x7f667815e4e0 2024-03-31T15:55:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6676000640 1 -- 172.21.15.90:0/1049055967 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66680ab600 con 0x7f667815e4e0 2024-03-31T15:55:37.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6676000640 1 -- 172.21.15.90:0/1049055967 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66680ab9d0 con 0x7f667815e4e0 2024-03-31T15:55:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- 172.21.15.90:0/1049055967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667815e4e0 msgr2=0x7f667806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 --2- 172.21.15.90:0/1049055967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667815e4e0 0x7f667806c910 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f66680097f0 tx=0x7f666801c410 comp rx=0 tx=0).stop 2024-03-31T15:55:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- 172.21.15.90:0/1049055967 shutdown_connections 2024-03-31T15:55:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 --2- 172.21.15.90:0/1049055967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667815e4e0 0x7f667806c910 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- 172.21.15.90:0/1049055967 >> 172.21.15.90:0/1049055967 conn(0x7f66780d1f20 msgr2=0x7f66780d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- 172.21.15.90:0/1049055967 shutdown_connections 2024-03-31T15:55:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- 172.21.15.90:0/1049055967 wait complete. 2024-03-31T15:55:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 Processor -- start 2024-03-31T15:55:37.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- start start 2024-03-31T15:55:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 0x7f66780ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 0x7f6678160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f66780d0d90 con 0x7f667806ce50 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f66780d0c10 con 0x7f66780cdf20 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f667e000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f667816c850 con 0x7f667815e4e0 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6677400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6677400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54604/0 (socket says 172.21.15.90:54604) 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.584+0000 7f6677400640 1 -- 172.21.15.90:0/1313352286 learned_addr learned my addr 172.21.15.90:0/1313352286 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 -- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 msgr2=0x7f6678160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6676a00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 0x7f66780ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677e00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 0x7f6678160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 0x7f6678160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 -- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 msgr2=0x7f66780ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 0x7f66780ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 -- 172.21.15.90:0/1313352286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667815fa60 con 0x7f667806ce50 2024-03-31T15:55:37.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677e00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 0x7f6678160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6676a00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 0x7f66780ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f6677400640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f66680ae040 tx=0x7f666801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66680b0050 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66680bc070 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66680b5a10 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667815fd40 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6678160300 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66780b7d20 con 0x7f667806ce50 2024-03-31T15:55:37.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f66680ab4c0 con 0x7f667806ce50 2024-03-31T15:55:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.588+0000 7f666f400640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 0x7f665806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.592+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f66681215e0 con 0x7f667806ce50 2024-03-31T15:55:37.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.592+0000 7f6676a00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 0x7f665806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.592+0000 7f6676a00640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 0x7f665806a840 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6678160c00 tx=0x7f6664009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.596+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6668016090 con 0x7f667806ce50 2024-03-31T15:55:37.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.760+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f66780685f0 con 0x7f667806ce50 2024-03-31T15:55:37.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.760+0000 7f666f400640 1 -- 172.21.15.90:0/1313352286 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f66680f17d0 con 0x7f667806ce50 2024-03-31T15:55:37.755 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 msgr2=0x7f665806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 0x7f665806a840 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6678160c00 tx=0x7f6664009040 comp rx=0 tx=0).stop 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 msgr2=0x7f6678152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f66680ae040 tx=0x7f666801ac50 comp rx=0 tx=0).stop 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 shutdown_connections 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6658068380 0x7f665806a840 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f667815e4e0 0x7f6678160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66780cdf20 0x7f66780ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 --2- 172.21.15.90:0/1313352286 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f667806ce50 0x7f6678152f10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 >> 172.21.15.90:0/1313352286 conn(0x7f66780d1f20 msgr2=0x7f66780d2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 shutdown_connections 2024-03-31T15:55:37.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.764+0000 7f667e000640 1 -- 172.21.15.90:0/1313352286 wait complete. 2024-03-31T15:55:37.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:55:37.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 Processor -- start 2024-03-31T15:55:37.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 -- start start 2024-03-31T15:55:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f78980d0860 0x7f78980b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789806cd00 con 0x7f789815e720 2024-03-31T15:55:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789806cfa0 con 0x7f789815e340 2024-03-31T15:55:37.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ee00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f78980b45c0 con 0x7f78980d0860 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ce00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f78980d0860 0x7f78980b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f7897400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f789815e720 0x7f78980d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:41780/0 (socket says 172.21.15.90:41780) 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f7897400640 1 -- 172.21.15.90:0/2884359893 learned_addr learned my addr 172.21.15.90:0/2884359893 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ce00640 1 -- 172.21.15.90:0/2884359893 >> v1:172.21.15.102:6789/0 conn(0x7f789815e340 legacy=0x7f789806da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ce00640 1 -- 172.21.15.90:0/2884359893 >> v1:172.21.15.90:6789/0 conn(0x7f789815e720 legacy=0x7f78980d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.920+0000 7f789ce00640 1 -- 172.21.15.90:0/2884359893 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78980b42e0 con 0x7f78980d0860 2024-03-31T15:55:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ce00640 1 --2- 172.21.15.90:0/2884359893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78980d0860 0x7f78980b1c80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7890009d90 tx=0x7f789001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbef6e4f98ac4537 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f7896a00640 1 -- 172.21.15.90:0/2884359893 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900b2070 con 0x7f78980d0860 2024-03-31T15:55:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f7896a00640 1 -- 172.21.15.90:0/2884359893 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78900a5940 con 0x7f78980d0860 2024-03-31T15:55:37.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f7896a00640 1 -- 172.21.15.90:0/2884359893 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900a5d10 con 0x7f78980d0860 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 -- 172.21.15.90:0/2884359893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78980d0860 msgr2=0x7f78980b1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 --2- 172.21.15.90:0/2884359893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78980d0860 0x7f78980b1c80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7890009d90 tx=0x7f789001c970 comp rx=0 tx=0).stop 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 -- 172.21.15.90:0/2884359893 shutdown_connections 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 --2- 172.21.15.90:0/2884359893 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f78980d0860 0x7f78980b1c80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 -- 172.21.15.90:0/2884359893 >> 172.21.15.90:0/2884359893 conn(0x7f789809aa40 msgr2=0x7f789809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 -- 172.21.15.90:0/2884359893 shutdown_connections 2024-03-31T15:55:37.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.924+0000 7f789ee00640 1 -- 172.21.15.90:0/2884359893 wait complete. 2024-03-31T15:55:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 Processor -- start 2024-03-31T15:55:37.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- start start 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78980d0860 0x7f78980c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 0x7f7898158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f78980b3f50 con 0x7f789815e720 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f789806e150 con 0x7f78980d0860 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f789816c210 con 0x7f789815e340 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54624/0 (socket says 172.21.15.90:54624) 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 -- 172.21.15.90:0/1595719159 learned_addr learned my addr 172.21.15.90:0/1595719159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f7897e00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78980d0860 0x7f78980c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f7897400640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 0x7f7898158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 -- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 msgr2=0x7f7898158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 0x7f7898158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 -- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78980d0860 msgr2=0x7f78980c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78980d0860 0x7f78980c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 -- 172.21.15.90:0/1595719159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78980c7230 con 0x7f789815e720 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f7897400640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 0x7f7898158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ce00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f789001a5a0 tx=0x7f7890019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900b2050 con 0x7f789815e720 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78900be070 con 0x7f789815e720 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78980c7510 con 0x7f789815e720 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900ada70 con 0x7f789815e720 2024-03-31T15:55:37.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.928+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78980c7ad0 con 0x7f789815e720 2024-03-31T15:55:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7898068660 con 0x7f789815e720 2024-03-31T15:55:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f78900adc10 con 0x7f789815e720 2024-03-31T15:55:37.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f788fe00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 0x7f787806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f7890121b00 con 0x7f789815e720 2024-03-31T15:55:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f7897e00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 0x7f787806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:37.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.932+0000 7f7897e00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 0x7f787806a870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f78980c8a80 tx=0x7f7888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:37.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:37.936+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7890016560 con 0x7f789815e720 2024-03-31T15:55:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7898152ac0 con 0x7f789815e720 2024-03-31T15:55:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f788fe00640 1 -- 172.21.15.90:0/1595719159 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f78900f1ea0 con 0x7f789815e720 2024-03-31T15:55:38.098 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:55:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 msgr2=0x7f787806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 0x7f787806a870 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f78980c8a80 tx=0x7f7888009040 comp rx=0 tx=0).stop 2024-03-31T15:55:38.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 msgr2=0x7f7898153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.104+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f789001a5a0 tx=0x7f7890019710 comp rx=0 tx=0).stop 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 shutdown_connections 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f78780683b0 0x7f787806a870 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f789815e720 0x7f7898153290 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f789815e340 0x7f7898158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 --2- 172.21.15.90:0/1595719159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78980d0860 0x7f78980c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 >> 172.21.15.90:0/1595719159 conn(0x7f789809aa40 msgr2=0x7f78980aeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 shutdown_connections 2024-03-31T15:55:38.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:38.108+0000 7f789ee00640 1 -- 172.21.15.90:0/1595719159 wait complete. 2024-03-31T15:55:38.119 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:55:38.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:38.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:38.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[38840]: fuse finished with error 0 and tester_r 0 2024-03-31T15:55:38.140 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:44.242 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:55:44.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.243 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:55:44.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:55:44.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 Processor -- start 2024-03-31T15:55:44.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 -- start start 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc18806b820 0x7fc18806bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1880b34f0 con 0x7fc1880cc9d0 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1880b37d0 con 0x7fc1880ccdb0 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18ec00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc18806c1d0 con 0x7fc18806b820 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc1880cc9d0 0x7fc188142030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56716/0 (socket says 172.21.15.90:56716) 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187e00640 1 -- 172.21.15.90:0/1523579288 learned_addr learned my addr 172.21.15.90:0/1523579288 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:44.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18cc00640 1 --2- 172.21.15.90:0/1523579288 >> v2:172.21.15.102:3300/0 conn(0x7fc18806b820 0x7fc18806bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319687139 0 0) 0x7fc1880b34f0 con 0x7fc1880cc9d0 2024-03-31T15:55:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc16c003530 con 0x7fc1880cc9d0 2024-03-31T15:55:44.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953490607 0 0) 0x7fc1880b37d0 con 0x7fc1880ccdb0 2024-03-31T15:55:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1880b34f0 con 0x7fc1880ccdb0 2024-03-31T15:55:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.436+0000 7fc18cc00640 1 -- 172.21.15.90:0/1523579288 >> v1:172.21.15.102:6789/0 conn(0x7fc1880ccdb0 legacy=0x7fc1880b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.440+0000 7fc18cc00640 1 -- 172.21.15.90:0/1523579288 >> v1:172.21.15.90:6789/0 conn(0x7fc1880cc9d0 legacy=0x7fc188142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.440+0000 7fc18cc00640 1 -- 172.21.15.90:0/1523579288 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc188141830 con 0x7fc18806b820 2024-03-31T15:55:44.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18cc00640 1 --2- 172.21.15.90:0/1523579288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc18806b820 0x7fc18806bc90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc180009d90 tx=0x7fc18001c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0b18074c80d7132 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800b2070 con 0x7fc18806b820 2024-03-31T15:55:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1800a58b0 con 0x7fc18806b820 2024-03-31T15:55:44.437 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc187400640 1 -- 172.21.15.90:0/1523579288 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800a5c80 con 0x7fc18806b820 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- 172.21.15.90:0/1523579288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc18806b820 msgr2=0x7fc18806bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 --2- 172.21.15.90:0/1523579288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc18806b820 0x7fc18806bc90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc180009d90 tx=0x7fc18001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- 172.21.15.90:0/1523579288 shutdown_connections 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 --2- 172.21.15.90:0/1523579288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc18806b820 0x7fc18806bc90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- 172.21.15.90:0/1523579288 >> 172.21.15.90:0/1523579288 conn(0x7fc1880b1c80 msgr2=0x7fc1880b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- 172.21.15.90:0/1523579288 shutdown_connections 2024-03-31T15:55:44.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- 172.21.15.90:0/1523579288 wait complete. 2024-03-31T15:55:44.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 Processor -- start 2024-03-31T15:55:44.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.444+0000 7fc18ec00640 1 -- start start 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc1880cc9d0 0x7fc18813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1880ccdb0 0x7fc18813d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc1880cf040 con 0x7fc18806b820 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc1880ceec0 con 0x7fc1880ccdb0 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc188141ba0 con 0x7fc1880cc9d0 2024-03-31T15:55:44.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51542/0 (socket says 172.21.15.90:51542) 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 -- 172.21.15.90:0/3398013972 learned_addr learned my addr 172.21.15.90:0/3398013972 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18cc00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1880ccdb0 0x7fc18813d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc17fe00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc1880cc9d0 0x7fc18813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 -- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc1880cc9d0 msgr2=0x7fc18813dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc1880cc9d0 0x7fc18813dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 -- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1880ccdb0 msgr2=0x7fc18813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1880ccdb0 0x7fc18813d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 -- 172.21.15.90:0/3398013972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18815ed60 con 0x7fc18806b820 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc187e00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c500 tx=0x7fc17800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178017070 con 0x7fc18806b820 2024-03-31T15:55:44.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc178005cf0 con 0x7fc18806b820 2024-03-31T15:55:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18815f040 con 0x7fc18806b820 2024-03-31T15:55:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178007330 con 0x7fc18806b820 2024-03-31T15:55:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc188073230 con 0x7fc18806b820 2024-03-31T15:55:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.448+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1880686d0 con 0x7fc18806b820 2024-03-31T15:55:44.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.452+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc17801a070 con 0x7fc18806b820 2024-03-31T15:55:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.452+0000 7fc184c00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 0x7fc16406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.452+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fc1780860c0 con 0x7fc18806b820 2024-03-31T15:55:44.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.452+0000 7fc17fe00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 0x7fc16406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:44.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.452+0000 7fc17fe00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 0x7fc16406a840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc170007fd0 tx=0x7fc170009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:44.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.456+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc178055cf0 con 0x7fc18806b820 2024-03-31T15:55:44.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.668+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc1880b4aa0 con 0x7fc18806b820 2024-03-31T15:55:44.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.668+0000 7fc184c00640 1 -- 172.21.15.90:0/3398013972 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+82 (secure 0 0 0) 0x7fc178059ff0 con 0x7fc18806b820 2024-03-31T15:55:44.662 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-31T15:55:44.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 msgr2=0x7fc16406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 0x7fc16406a840 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc170007fd0 tx=0x7fc170009040 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 msgr2=0x7fc188140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c500 tx=0x7fc17800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 shutdown_connections 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc164068380 0x7fc16406a840 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1880ccdb0 0x7fc18813d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc1880cc9d0 0x7fc18813dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 --2- 172.21.15.90:0/3398013972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc18806b820 0x7fc188140810 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 >> 172.21.15.90:0/3398013972 conn(0x7fc1880b1c80 msgr2=0x7fc188142030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 shutdown_connections 2024-03-31T15:55:44.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44.672+0000 7fc18ec00640 1 -- 172.21.15.90:0/3398013972 wait complete. 2024-03-31T15:55:44.684 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:55:44.684 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:55:44.684 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:55:44.685 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:44.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.685 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:55:44.694 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:55:44.694 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.695 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42379sec preferred_lft 42379sec 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:44.696 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:44.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.697 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:55:44.753 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:55:44.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.756 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:44.812 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:55:44.813 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:55:44.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:55:44.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:44.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.953 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:55:44.961 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:55:44.961 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:55:44.962 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:44.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:44.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:55:45.023 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:55:45.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:45.023 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:45.072 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:45.073 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:45.073 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:45.123 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:45.183 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:45.233 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:45.234 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:45.235 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:45.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.236 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:45.287 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:55:45.287 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:55:45.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:45.351 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:45.404 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:45.406 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:45.407 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:45.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.408 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:45.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:55:45.428+0000 7fafc9a4e580 -1 init, newargv = 0x55877bd440c0 newargc=15 2024-03-31T15:55:45.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39400]: starting ceph client 2024-03-31T15:55:45.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39400]: starting fuse 2024-03-31T15:55:45.433 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:55:45.433 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:55:45.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:55:45.433 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:55:45.433 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:45.434 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:55:45.435 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:55:45.556 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.39400.asok 2024-03-31T15:55:45.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:45.567 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39400.asok 2024-03-31T15:55:45.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39400.asok status 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:55:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "39400", 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5868, 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 5868 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3530116830 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3530116830 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.5868 192.168.0.1:0/3530116830", 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/3530116830", 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 18, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 43, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:55:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:55:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:55:45.740 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:55:45.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.752 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:55:45.752 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:45.763 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:55:45.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:45.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:45.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.827 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-31T15:55:45.877 INFO:teuthology.orchestra.run.smithi090.stderr:setfattr: layoutfile: Permission denied 2024-03-31T15:55:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:55:45.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:45.879 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-31T15:55:45.929 INFO:teuthology.orchestra.run.smithi090.stderr:setfattr: layoutdir: Permission denied 2024-03-31T15:55:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:55:45.930 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:55:45.930 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:55:45.980 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:46.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.160+0000 7fb91fc00640 1 Processor -- start 2024-03-31T15:55:46.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.160+0000 7fb91fc00640 1 -- start start 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb91815c120 0x7fb91815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9180d2680 0x7fb9180b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180b2c20 0x7fb9180cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb918160bf0 con 0x7fb91815c120 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb9180d2b10 con 0x7fb9180d2680 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91fc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb918160350 con 0x7fb9180b2c20 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb91815c120 0x7fb91815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb91815c120 0x7fb91815fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51554/0 (socket says 172.21.15.90:51554) 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 -- 172.21.15.90:0/2720249993 learned_addr learned my addr 172.21.15.90:0/2720249993 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:46.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb917e00640 1 --2- 172.21.15.90:0/2720249993 >> v2:172.21.15.102:3300/0 conn(0x7fb9180d2680 0x7fb9180b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91dc00640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180b2c20 0x7fb9180cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 -- 172.21.15.90:0/2720249993 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180b2c20 msgr2=0x7fb9180cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180b2c20 0x7fb9180cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 -- 172.21.15.90:0/2720249993 >> v2:172.21.15.102:3300/0 conn(0x7fb9180d2680 msgr2=0x7fb9180b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 --2- 172.21.15.90:0/2720249993 >> v2:172.21.15.102:3300/0 conn(0x7fb9180d2680 0x7fb9180b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.164+0000 7fb91d200640 1 -- 172.21.15.90:0/2720249993 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb918160630 con 0x7fb91815c120 2024-03-31T15:55:46.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91d200640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb91815fd00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb91806ed50 tx=0x7fb90c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c145a27880f69a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb917400640 1 -- 172.21.15.90:0/2720249993 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0b0070 con 0x7fb91815c120 2024-03-31T15:55:46.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb917400640 1 -- 172.21.15.90:0/2720249993 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb90c0ab600 con 0x7fb91815c120 2024-03-31T15:55:46.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb917400640 1 -- 172.21.15.90:0/2720249993 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0ab9d0 con 0x7fb91815c120 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- 172.21.15.90:0/2720249993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 msgr2=0x7fb91815fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb91815fd00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb91806ed50 tx=0x7fb90c01c3d0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- 172.21.15.90:0/2720249993 shutdown_connections 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180b2c20 0x7fb9180cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- 172.21.15.90:0/2720249993 >> v2:172.21.15.102:3300/0 conn(0x7fb9180d2680 0x7fb9180b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- 172.21.15.90:0/2720249993 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb91815fd00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- 172.21.15.90:0/2720249993 >> 172.21.15.90:0/2720249993 conn(0x7fb91809aa40 msgr2=0x7fb91809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:46.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- 172.21.15.90:0/2720249993 shutdown_connections 2024-03-31T15:55:46.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- 172.21.15.90:0/2720249993 wait complete. 2024-03-31T15:55:46.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 Processor -- start 2024-03-31T15:55:46.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- start start 2024-03-31T15:55:46.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9180b2c20 0x7fb91806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180d2680 0x7fb918158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180cea90 con 0x7fb91815c120 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180ce910 con 0x7fb9180b2c20 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.168+0000 7fb91fc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb9180cec10 con 0x7fb9180d2680 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51558/0 (socket says 172.21.15.90:51558) 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 -- 172.21.15.90:0/1824446492 learned_addr learned my addr 172.21.15.90:0/1824446492 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91d200640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9180b2c20 0x7fb91806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb917e00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180d2680 0x7fb918158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 -- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180d2680 msgr2=0x7fb918158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180d2680 0x7fb918158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 -- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9180b2c20 msgr2=0x7fb91806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9180b2c20 0x7fb91806c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 -- 172.21.15.90:0/1824446492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91806d1b0 con 0x7fb91815c120 2024-03-31T15:55:46.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91dc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb91000eed0 tx=0x7fb91000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910018020 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb910014400 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91806d4f0 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9100147f0 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91806dd30 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91806df30 con 0x7fb91815c120 2024-03-31T15:55:46.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.172+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb91001b020 con 0x7fb91815c120 2024-03-31T15:55:46.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.176+0000 7fb914c00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 0x7fb8f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.176+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fb9100873c0 con 0x7fb91815c120 2024-03-31T15:55:46.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.176+0000 7fb91d200640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 0x7fb8f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.176+0000 7fb91d200640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 0x7fb8f806a840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0ad000 tx=0x7fb90c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.180+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb91806df30 con 0x7fb91815c120 2024-03-31T15:55:46.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.340+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9180686d0 con 0x7fb91815c120 2024-03-31T15:55:46.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb914c00640 1 -- 172.21.15.90:0/1824446492 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7fb910022070 con 0x7fb91815c120 2024-03-31T15:55:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:46.337 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 msgr2=0x7fb8f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 0x7fb8f806a840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0ad000 tx=0x7fb90c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 msgr2=0x7fb918150cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb91000eed0 tx=0x7fb91000b480 comp rx=0 tx=0).stop 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 shutdown_connections 2024-03-31T15:55:46.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb8f8068380 0x7fb8f806a840 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb91815c120 0x7fb918150cf0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb9180d2680 0x7fb918158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 --2- 172.21.15.90:0/1824446492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9180b2c20 0x7fb91806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 >> 172.21.15.90:0/1824446492 conn(0x7fb91809aa40 msgr2=0x7fb918157d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 shutdown_connections 2024-03-31T15:55:46.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.344+0000 7fb91fc00640 1 -- 172.21.15.90:0/1824446492 wait complete. 2024-03-31T15:55:46.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:46.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.504+0000 7faafbe00640 1 Processor -- start 2024-03-31T15:55:46.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.504+0000 7faafbe00640 1 -- start start 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faafbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faaf415c320 0x7faaf40cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faafbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf40cd640 0x7faaf40af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faafbe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaf40cefc0 con 0x7faaf40d2b20 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faafbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7faaf40aed70 con 0x7faaf415c320 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faafbe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7faaf40cf2a0 con 0x7faaf40cd640 2024-03-31T15:55:46.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf8a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7faaf40d2b20 0x7faaf40ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56724/0 (socket says 172.21.15.90:56724) 2024-03-31T15:55:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf8a00640 1 -- 172.21.15.90:0/1962732724 learned_addr learned my addr 172.21.15.90:0/1962732724 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9400640 1 --2- 172.21.15.90:0/1962732724 >> v2:172.21.15.102:3300/0 conn(0x7faaf415c320 0x7faaf40cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9e00640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf40cd640 0x7faaf40af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf3e00640 1 -- 172.21.15.90:0/1962732724 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1240715481 0 0) 0x7faaf40cefc0 con 0x7faaf40d2b20 2024-03-31T15:55:46.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf3e00640 1 -- 172.21.15.90:0/1962732724 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faadc003530 con 0x7faaf40d2b20 2024-03-31T15:55:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9400640 1 -- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf40cd640 msgr2=0x7faaf40af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9400640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf40cd640 0x7faaf40af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9400640 1 -- 172.21.15.90:0/1962732724 >> v1:172.21.15.90:6789/0 conn(0x7faaf40d2b20 legacy=0x7faaf40ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.508+0000 7faaf9400640 1 -- 172.21.15.90:0/1962732724 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf40cf580 con 0x7faaf415c320 2024-03-31T15:55:46.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faaf9400640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf415c320 0x7faaf40cc9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faae80097c0 tx=0x7faae801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d6944ce9eb4fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faaf3e00640 1 -- 172.21.15.90:0/1962732724 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80b0070 con 0x7faaf415c320 2024-03-31T15:55:46.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faaf3e00640 1 -- 172.21.15.90:0/1962732724 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf40d0f20 con 0x7faaf415c320 2024-03-31T15:55:46.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faaf3e00640 1 -- 172.21.15.90:0/1962732724 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80ab800 con 0x7faaf415c320 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf415c320 msgr2=0x7faaf40cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf415c320 0x7faaf40cc9f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faae80097c0 tx=0x7faae801c410 comp rx=0 tx=0).stop 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- 172.21.15.90:0/1962732724 shutdown_connections 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf40cd640 0x7faaf40af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 --2- 172.21.15.90:0/1962732724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf415c320 0x7faaf40cc9f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- 172.21.15.90:0/1962732724 >> 172.21.15.90:0/1962732724 conn(0x7faaf40d1f20 msgr2=0x7faaf40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- 172.21.15.90:0/1962732724 shutdown_connections 2024-03-31T15:55:46.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- 172.21.15.90:0/1962732724 wait complete. 2024-03-31T15:55:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 Processor -- start 2024-03-31T15:55:46.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.512+0000 7faafbe00640 1 -- start start 2024-03-31T15:55:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf40d2b20 0x7faaf40cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf415c320 0x7faaf4153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faaf40b0320 con 0x7faaf40cd640 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faaf40b01a0 con 0x7faaf40d2b20 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7faaf40b04a0 con 0x7faaf415c320 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51560/0 (socket says 172.21.15.90:51560) 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 -- 172.21.15.90:0/1171752757 learned_addr learned my addr 172.21.15.90:0/1171752757 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9e00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf415c320 0x7faaf4153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 -- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf415c320 msgr2=0x7faaf4153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf415c320 0x7faaf4153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 -- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf40d2b20 msgr2=0x7faaf40cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf40d2b20 0x7faaf40cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 -- 172.21.15.90:0/1171752757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf415f770 con 0x7faaf40cd640 2024-03-31T15:55:46.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf9400640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faae80ae040 tx=0x7faae801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80b0050 con 0x7faaf40cd640 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae80b1040 con 0x7faaf40cd640 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8006e20 con 0x7faaf40cd640 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf415f490 con 0x7faaf40cd640 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf41595a0 con 0x7faaf40cd640 2024-03-31T15:55:46.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf40685f0 con 0x7faaf40cd640 2024-03-31T15:55:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.516+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faae80be030 con 0x7faaf40cd640 2024-03-31T15:55:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.520+0000 7faaf1600640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 0x7faad406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.520+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7faae8120a40 con 0x7faaf40cd640 2024-03-31T15:55:46.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.520+0000 7faaf8a00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 0x7faad406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.520+0000 7faaf8a00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 0x7faad406a840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faae0007fd0 tx=0x7faae0007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.524+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7faae80c5020 con 0x7faaf40cd640 2024-03-31T15:55:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.680+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faaf415bb50 con 0x7faaf40cd640 2024-03-31T15:55:46.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.680+0000 7faaf1600640 1 -- 172.21.15.90:0/1171752757 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7faae8016090 con 0x7faaf40cd640 2024-03-31T15:55:46.676 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 msgr2=0x7faad406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 0x7faad406a840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faae0007fd0 tx=0x7faae0007900 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 msgr2=0x7faaf40cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faae80ae040 tx=0x7faae801ac50 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 shutdown_connections 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7faad4068380 0x7faad406a840 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faaf415c320 0x7faaf4153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faaf40d2b20 0x7faaf40cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 --2- 172.21.15.90:0/1171752757 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faaf40cd640 0x7faaf40cae90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 >> 172.21.15.90:0/1171752757 conn(0x7faaf40d1f20 msgr2=0x7faaf4158990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:46.678 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 shutdown_connections 2024-03-31T15:55:46.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.684+0000 7faafbe00640 1 -- 172.21.15.90:0/1171752757 wait complete. 2024-03-31T15:55:46.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 Processor -- start 2024-03-31T15:55:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 -- start start 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38380cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f38380cd640 0x7f38380af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38380cefc0 con 0x7f383815c320 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f38380aed70 con 0x7f38380cd640 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3840400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f38380cf2a0 con 0x7f38380d2b20 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383da00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f383815c320 0x7f38380cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56732/0 (socket says 172.21.15.90:56732) 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383da00640 1 -- 172.21.15.90:0/885690941 learned_addr learned my addr 172.21.15.90:0/885690941 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:46.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383e400640 1 --2- 172.21.15.90:0/885690941 >> v2:172.21.15.90:3300/0 conn(0x7f38380cd640 0x7f38380af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3837e00640 1 -- 172.21.15.90:0/885690941 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354156336 0 0) 0x7f38380cefc0 con 0x7f383815c320 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383d000640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38380cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f3837e00640 1 -- 172.21.15.90:0/885690941 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3820003530 con 0x7f383815c320 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383e400640 1 -- 172.21.15.90:0/885690941 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 msgr2=0x7f38380cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383e400640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38380cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383e400640 1 -- 172.21.15.90:0/885690941 >> v1:172.21.15.90:6789/0 conn(0x7f383815c320 legacy=0x7f38380cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383e400640 1 -- 172.21.15.90:0/885690941 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38380cf580 con 0x7f38380cd640 2024-03-31T15:55:46.840 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.844+0000 7f383d000640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38380cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:46.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f383e400640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f38380af200 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3830009d90 tx=0x7f383001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968813bfa57319ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3837e00640 1 -- 172.21.15.90:0/885690941 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38300b2070 con 0x7f38380cd640 2024-03-31T15:55:46.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3837e00640 1 -- 172.21.15.90:0/885690941 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38380d0f20 con 0x7f38380cd640 2024-03-31T15:55:46.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3837e00640 1 -- 172.21.15.90:0/885690941 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38300a5ad0 con 0x7f38380cd640 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 -- 172.21.15.90:0/885690941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 msgr2=0x7f38380af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f38380af200 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3830009d90 tx=0x7f383001c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 -- 172.21.15.90:0/885690941 shutdown_connections 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f38380af200 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 --2- 172.21.15.90:0/885690941 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38380cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 -- 172.21.15.90:0/885690941 >> 172.21.15.90:0/885690941 conn(0x7f38380d1f20 msgr2=0x7f38380d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 -- 172.21.15.90:0/885690941 shutdown_connections 2024-03-31T15:55:46.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.848+0000 7f3840400640 1 -- 172.21.15.90:0/885690941 wait complete. 2024-03-31T15:55:46.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 Processor -- start 2024-03-31T15:55:46.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 -- start start 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38381541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383815c320 0x7f3838156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f38380b0360 con 0x7f38380cd640 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38380b01e0 con 0x7f383815c320 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f3840400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f38380b04e0 con 0x7f38380d2b20 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f383da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f383da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51588/0 (socket says 172.21.15.90:51588) 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f383da00640 1 -- 172.21.15.90:0/608755657 learned_addr learned my addr 172.21.15.90:0/608755657 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:46.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f383d000640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38381541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.852+0000 7f383e400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383815c320 0x7f3838156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 -- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 msgr2=0x7f38381541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38381541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 -- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383815c320 msgr2=0x7f3838156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383815c320 0x7f3838156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 -- 172.21.15.90:0/608755657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38381554b0 con 0x7f38380cd640 2024-03-31T15:55:46.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383da00640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c450 tx=0x7f382c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c017070 con 0x7f38380cd640 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382c005cf0 con 0x7f38380cd640 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c007290 con 0x7f38380cd640 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3838154c10 con 0x7f38380cd640 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38381551d0 con 0x7f38380cd640 2024-03-31T15:55:46.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38380685f0 con 0x7f38380cd640 2024-03-31T15:55:46.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f382c01a070 con 0x7f38380cd640 2024-03-31T15:55:46.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 0x7f381806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:46.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f382c086020 con 0x7f38380cd640 2024-03-31T15:55:46.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.856+0000 7f383d000640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 0x7f381806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:46.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.860+0000 7f383d000640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 0x7f381806a840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f38380c8ca0 tx=0x7f3824008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:46.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:46.860+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f382c055c50 con 0x7f38380cd640 2024-03-31T15:55:47.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3838071380 con 0x7f38380cd640 2024-03-31T15:55:47.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3835600640 1 -- 172.21.15.90:0/608755657 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f382c059f50 con 0x7f38380cd640 2024-03-31T15:55:47.013 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 msgr2=0x7f381806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 0x7f381806a840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f38380c8ca0 tx=0x7f3824008040 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 msgr2=0x7f3838146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c450 tx=0x7f382c00c910 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 shutdown_connections 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3818068380 0x7f381806a840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383815c320 0x7f3838156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38380d2b20 0x7f38381541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 --2- 172.21.15.90:0/608755657 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38380cd640 0x7f3838146120 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 >> 172.21.15.90:0/608755657 conn(0x7f38380d1f20 msgr2=0x7f3838070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 shutdown_connections 2024-03-31T15:55:47.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.020+0000 7f3840400640 1 -- 172.21.15.90:0/608755657 wait complete. 2024-03-31T15:55:47.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:47.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 Processor -- start 2024-03-31T15:55:47.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 -- start start 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 0x7f43b4161e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b40d0b50 con 0x7f43b415e3e0 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b4162510 con 0x7f43b40cbb70 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b9200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43b4162db0 con 0x7f43b40706c0 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.180+0000 7f43b1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f43b415e3e0 0x7f43b40cb460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56734/0 (socket says 172.21.15.90:56734) 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1e00640 1 -- 172.21.15.90:0/2780745112 learned_addr learned my addr 172.21.15.90:0/2780745112 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b2800640 1 --2- 172.21.15.90:0/2780745112 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 0x7f43b4161e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728567675 0 0) 0x7f43b40d0b50 con 0x7f43b415e3e0 2024-03-31T15:55:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4398003530 con 0x7f43b415e3e0 2024-03-31T15:55:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b2800640 1 -- 172.21.15.90:0/2780745112 >> v1:172.21.15.102:6789/0 conn(0x7f43b40cbb70 legacy=0x7f43b40cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614279885 0 0) 0x7f43b4162510 con 0x7f43b40cbb70 2024-03-31T15:55:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b2800640 1 -- 172.21.15.90:0/2780745112 >> v1:172.21.15.90:6789/0 conn(0x7f43b415e3e0 legacy=0x7f43b40cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b2800640 1 -- 172.21.15.90:0/2780745112 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b41627f0 con 0x7f43b40706c0 2024-03-31T15:55:47.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b2800640 1 --2- 172.21.15.90:0/2780745112 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 0x7f43b4161e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009dc0 tx=0x7f43ac01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea24bb5dfa83057b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0b2070 con 0x7f43b40706c0 2024-03-31T15:55:47.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ac0a59c0 con 0x7f43b40706c0 2024-03-31T15:55:47.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.184+0000 7f43b1400640 1 -- 172.21.15.90:0/2780745112 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0a5d90 con 0x7f43b40706c0 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- 172.21.15.90:0/2780745112 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 msgr2=0x7f43b4161e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 --2- 172.21.15.90:0/2780745112 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 0x7f43b4161e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009dc0 tx=0x7f43ac01c900 comp rx=0 tx=0).stop 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- 172.21.15.90:0/2780745112 shutdown_connections 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 --2- 172.21.15.90:0/2780745112 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40706c0 0x7f43b4161e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- 172.21.15.90:0/2780745112 >> 172.21.15.90:0/2780745112 conn(0x7f43b40cde90 msgr2=0x7f43b40ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- 172.21.15.90:0/2780745112 shutdown_connections 2024-03-31T15:55:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- 172.21.15.90:0/2780745112 wait complete. 2024-03-31T15:55:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 Processor -- start 2024-03-31T15:55:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- start start 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40706c0 0x7f43b40c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40cbb70 0x7f43b4153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f43b40d0360 con 0x7f43b415e3e0 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43b40d01e0 con 0x7f43b40706c0 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b9200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f43b416c1c0 con 0x7f43b40cbb70 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51604/0 (socket says 172.21.15.90:51604) 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 -- 172.21.15.90:0/1586927142 learned_addr learned my addr 172.21.15.90:0/1586927142 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43abe00640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40cbb70 0x7f43b4153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b1e00640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40706c0 0x7f43b40c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 -- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40cbb70 msgr2=0x7f43b4153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40cbb70 0x7f43b4153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 -- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40706c0 msgr2=0x7f43b40c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40706c0 0x7f43b40c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.188+0000 7f43b2800640 1 -- 172.21.15.90:0/1586927142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b4154ba0 con 0x7f43b415e3e0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43b2800640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0b0040 tx=0x7f43ac0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0b2050 con 0x7f43b415e3e0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ac0b3040 con 0x7f43b415e3e0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac0ad850 con 0x7f43b415e3e0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b40c9b80 con 0x7f43b415e3e0 2024-03-31T15:55:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b40ca140 con 0x7f43b415e3e0 2024-03-31T15:55:47.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b40685f0 con 0x7f43b415e3e0 2024-03-31T15:55:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f43ac0c0020 con 0x7f43b415e3e0 2024-03-31T15:55:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 0x7f439006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f43ac121490 con 0x7f43b415e3e0 2024-03-31T15:55:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.192+0000 7f43b1e00640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 0x7f439006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.196+0000 7f43b1e00640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 0x7f439006a890 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007fd0 tx=0x7f43a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.196+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f43ac016580 con 0x7f43b415e3e0 2024-03-31T15:55:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43b4161e30 con 0x7f43b415e3e0 2024-03-31T15:55:47.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43aa000640 1 -- 172.21.15.90:0/1586927142 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f43ac0f10c0 con 0x7f43b415e3e0 2024-03-31T15:55:47.361 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 msgr2=0x7f439006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 0x7f439006a890 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007fd0 tx=0x7f43a4009040 comp rx=0 tx=0).stop 2024-03-31T15:55:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 msgr2=0x7f43b415ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f43ac0b0040 tx=0x7f43ac0196a0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 shutdown_connections 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f43900683d0 0x7f439006a890 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f43b415e3e0 0x7f43b415ea70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f43b40cbb70 0x7f43b4153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 --2- 172.21.15.90:0/1586927142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43b40706c0 0x7f43b40c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 >> 172.21.15.90:0/1586927142 conn(0x7f43b40cde90 msgr2=0x7f43b40d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 shutdown_connections 2024-03-31T15:55:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.368+0000 7f43b9200640 1 -- 172.21.15.90:0/1586927142 wait complete. 2024-03-31T15:55:47.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:55:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 Processor -- start 2024-03-31T15:55:47.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 -- start start 2024-03-31T15:55:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa9200cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9200d2e60 0x7fa9200b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9200b71d0 con 0x7fa9200cfc80 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa9200d00c0 con 0x7fa9200d2e60 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa927a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa9200b74b0 con 0x7fa92015e6a0 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa91fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa9200cfc80 0x7fa9200d2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56748/0 (socket says 172.21.15.90:56748) 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa91fe00640 1 -- 172.21.15.90:0/3034882033 learned_addr learned my addr 172.21.15.90:0/3034882033 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:47.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925a00640 1 --2- 172.21.15.90:0/3034882033 >> v2:172.21.15.102:3300/0 conn(0x7fa9200d2e60 0x7fa9200b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925000640 1 --2- 172.21.15.90:0/3034882033 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa9200cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa91f400640 1 -- 172.21.15.90:0/3034882033 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 89893908 0 0) 0x7fa9200b71d0 con 0x7fa9200cfc80 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa91f400640 1 -- 172.21.15.90:0/3034882033 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa908003530 con 0x7fa9200cfc80 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925000640 1 -- 172.21.15.90:0/3034882033 >> v2:172.21.15.102:3300/0 conn(0x7fa9200d2e60 msgr2=0x7fa9200b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925000640 1 --2- 172.21.15.90:0/3034882033 >> v2:172.21.15.102:3300/0 conn(0x7fa9200d2e60 0x7fa9200b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925000640 1 -- 172.21.15.90:0/3034882033 >> v1:172.21.15.90:6789/0 conn(0x7fa9200cfc80 legacy=0x7fa9200d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.528+0000 7fa925000640 1 -- 172.21.15.90:0/3034882033 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9200b7790 con 0x7fa92015e6a0 2024-03-31T15:55:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa925000640 1 --2- 172.21.15.90:0/3034882033 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa9200cf740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9140097c0 tx=0x7fa91401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21a2519fc8b0dc67 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa91f400640 1 -- 172.21.15.90:0/3034882033 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b5070 con 0x7fa92015e6a0 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa91f400640 1 -- 172.21.15.90:0/3034882033 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140a8990 con 0x7fa92015e6a0 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa91f400640 1 -- 172.21.15.90:0/3034882033 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140a8d60 con 0x7fa92015e6a0 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 -- 172.21.15.90:0/3034882033 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 msgr2=0x7fa9200cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 --2- 172.21.15.90:0/3034882033 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa9200cf740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9140097c0 tx=0x7fa91401ec90 comp rx=0 tx=0).stop 2024-03-31T15:55:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 -- 172.21.15.90:0/3034882033 shutdown_connections 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 --2- 172.21.15.90:0/3034882033 >> v2:172.21.15.102:3300/0 conn(0x7fa9200d2e60 0x7fa9200b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 --2- 172.21.15.90:0/3034882033 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa9200cf740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 -- 172.21.15.90:0/3034882033 >> 172.21.15.90:0/3034882033 conn(0x7fa92009d4b0 msgr2=0x7fa9200d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 -- 172.21.15.90:0/3034882033 shutdown_connections 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.532+0000 7fa927a00640 1 -- 172.21.15.90:0/3034882033 wait complete. 2024-03-31T15:55:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 Processor -- start 2024-03-31T15:55:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- start start 2024-03-31T15:55:47.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9200cfc80 0x7fa92015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa92006c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa9200b5320 con 0x7fa9200d2e60 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa9200b7d50 con 0x7fa9200cfc80 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa9200b54a0 con 0x7fa92015e6a0 2024-03-31T15:55:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51608/0 (socket says 172.21.15.90:51608) 2024-03-31T15:55:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 -- 172.21.15.90:0/2891503364 learned_addr learned my addr 172.21.15.90:0/2891503364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa925000640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9200cfc80 0x7fa92015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa925a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa92006c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 -- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 msgr2=0x7fa92006c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa92006c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 -- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9200cfc80 msgr2=0x7fa92015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9200cfc80 0x7fa92015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 -- 172.21.15.90:0/2891503364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92015b910 con 0x7fa9200d2e60 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91fe00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c450 tx=0x7fa90c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c017070 con 0x7fa9200d2e60 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c005cf0 con 0x7fa9200d2e60 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c007290 con 0x7fa9200d2e60 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9200c7e40 con 0x7fa9200d2e60 2024-03-31T15:55:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.536+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9200c8400 con 0x7fa9200d2e60 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9200685f0 con 0x7fa9200d2e60 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa90c01a070 con 0x7fa9200d2e60 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa91cc00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 0x7fa90006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fa90c085f80 con 0x7fa9200d2e60 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa925000640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 0x7fa90006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.540+0000 7fa925000640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 0x7fa90006a870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b2000 tx=0x7fa914008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.544+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa90c055c30 con 0x7fa9200d2e60 2024-03-31T15:55:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fa9200d2530 con 0x7fa9200d2e60 2024-03-31T15:55:47.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa91cc00640 1 -- 172.21.15.90:0/2891503364 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7fa90c059f30 con 0x7fa9200d2e60 2024-03-31T15:55:47.697 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 msgr2=0x7fa90006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 0x7fa90006a870 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b2000 tx=0x7fa914008040 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 msgr2=0x7fa92006c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c450 tx=0x7fa90c00c910 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 shutdown_connections 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa9000683b0 0x7fa90006a870 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa92015e6a0 0x7fa92006c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa9200d2e60 0x7fa92006c3f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 --2- 172.21.15.90:0/2891503364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9200cfc80 0x7fa92015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 >> 172.21.15.90:0/2891503364 conn(0x7fa92009d4b0 msgr2=0x7fa92006dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 shutdown_connections 2024-03-31T15:55:47.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.704+0000 7fa927a00640 1 -- 172.21.15.90:0/2891503364 wait complete. 2024-03-31T15:55:47.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:55:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.872+0000 7f990c400640 1 Processor -- start 2024-03-31T15:55:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.872+0000 7f990c400640 1 -- start start 2024-03-31T15:55:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f990415e320 0x7f99040d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 0x7f9904161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99041622d0 0x7f990406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f990415e700 con 0x7f990415e320 2024-03-31T15:55:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f99040d2dc0 con 0x7f99041622d0 2024-03-31T15:55:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990c400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f99040d1400 con 0x7f99040d2940 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f990415e320 0x7f99040d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f990415e320 0x7f99040d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51610/0 (socket says 172.21.15.90:51610) 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 -- 172.21.15.90:0/1556945319 learned_addr learned my addr 172.21.15.90:0/1556945319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909000640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 0x7f9904161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f990a400640 1 --2- 172.21.15.90:0/1556945319 >> v2:172.21.15.102:3300/0 conn(0x7f99041622d0 0x7f990406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 -- 172.21.15.90:0/1556945319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 msgr2=0x7f9904161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 0x7f9904161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 -- 172.21.15.90:0/1556945319 >> v2:172.21.15.102:3300/0 conn(0x7f99041622d0 msgr2=0x7f990406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 --2- 172.21.15.90:0/1556945319 >> v2:172.21.15.102:3300/0 conn(0x7f99041622d0 0x7f990406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 -- 172.21.15.90:0/1556945319 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99040b1a20 con 0x7f990415e320 2024-03-31T15:55:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909000640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 0x7f9904161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9909a00640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f990415e320 0x7f99040d0d20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f99040b7fb0 tx=0x7f98f801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3efbc9a31049e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.876+0000 7f9903e00640 1 -- 172.21.15.90:0/1556945319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f80b0070 con 0x7f990415e320 2024-03-31T15:55:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9903e00640 1 -- 172.21.15.90:0/1556945319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f80ab600 con 0x7f990415e320 2024-03-31T15:55:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9903e00640 1 -- 172.21.15.90:0/1556945319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f80ab9d0 con 0x7f990415e320 2024-03-31T15:55:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- 172.21.15.90:0/1556945319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f990415e320 msgr2=0x7f99040d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f990415e320 0x7f99040d0d20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f99040b7fb0 tx=0x7f98f801c260 comp rx=0 tx=0).stop 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- 172.21.15.90:0/1556945319 shutdown_connections 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- 172.21.15.90:0/1556945319 >> v2:172.21.15.102:3300/0 conn(0x7f99041622d0 0x7f990406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f99040d2940 0x7f9904161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- 172.21.15.90:0/1556945319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f990415e320 0x7f99040d0d20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- 172.21.15.90:0/1556945319 >> 172.21.15.90:0/1556945319 conn(0x7f990408bfd0 msgr2=0x7f99040d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- 172.21.15.90:0/1556945319 shutdown_connections 2024-03-31T15:55:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- 172.21.15.90:0/1556945319 wait complete. 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 Processor -- start 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- start start 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99040d2940 0x7f99041544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 0x7f990415e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f99040b2890 con 0x7f99041622d0 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99040b2710 con 0x7f99040d2940 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990c400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f99040b2a10 con 0x7f990415e320 2024-03-31T15:55:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f990a400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 0x7f990415e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9909000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9909a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99040d2940 0x7f99041544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9909000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:33676/0 (socket says 172.21.15.90:33676) 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.880+0000 7f9909000640 1 -- 172.21.15.90:0/196248323 learned_addr learned my addr 172.21.15.90:0/196248323 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 -- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 msgr2=0x7f99040b5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 -- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99040d2940 msgr2=0x7f99041544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99040d2940 0x7f99041544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:47.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 -- 172.21.15.90:0/196248323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99040b6360 con 0x7f99041622d0 2024-03-31T15:55:47.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9909000640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:47.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990a400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 0x7f990415e890 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f98fc007cf0 tx=0x7f98fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00edd0 con 0x7f99041622d0 2024-03-31T15:55:47.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc0039a0 con 0x7f99041622d0 2024-03-31T15:55:47.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99040b6640 con 0x7f99041622d0 2024-03-31T15:55:47.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003d90 con 0x7f99041622d0 2024-03-31T15:55:47.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99040b6c00 con 0x7f99041622d0 2024-03-31T15:55:47.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9904068660 con 0x7f99041622d0 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98fc01b020 con 0x7f99041622d0 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 0x7f98e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.884+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f98fc086d50 con 0x7f99041622d0 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.888+0000 7f9909a00640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 0x7f98e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.888+0000 7f9909a00640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 0x7f98e406a870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f98f80ad000 tx=0x7f98f80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:47.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:47.888+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f98fc0568f0 con 0x7f99041622d0 2024-03-31T15:55:47.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:55:47.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:55:48.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.072+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9904154a10 con 0x7f99041622d0 2024-03-31T15:55:48.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.072+0000 7f9901600640 1 -- 172.21.15.90:0/196248323 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f98fc05abf0 con 0x7f99041622d0 2024-03-31T15:55:48.067 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:55:48.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 msgr2=0x7f98e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 0x7f98e406a870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f98f80ad000 tx=0x7f98f80a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 msgr2=0x7f990415e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 0x7f990415e890 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f98fc007cf0 tx=0x7f98fc00e4b0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 shutdown_connections 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98e40683b0 0x7f98e406a870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f99041622d0 0x7f990415e890 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f990415e320 0x7f99040b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 --2- 172.21.15.90:0/196248323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99040d2940 0x7f99041544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 >> 172.21.15.90:0/196248323 conn(0x7f990408bfd0 msgr2=0x7f99040aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 shutdown_connections 2024-03-31T15:55:48.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.076+0000 7f990c400640 1 -- 172.21.15.90:0/196248323 wait complete. 2024-03-31T15:55:48.089 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:55:48.089 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:55:48.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:48.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 Processor -- start 2024-03-31T15:55:48.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 -- start start 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f020c15e1b0 0x7f020c162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 0x7f020c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020c15f780 con 0x7f020c06c7b0 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f020c1622b0 con 0x7f020c15e1b0 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f0211c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f020c15f1c0 con 0x7f020c06d7c0 2024-03-31T15:55:48.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f020aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f020c15e1b0 0x7f020c162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f020aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f020c15e1b0 0x7f020c162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51618/0 (socket says 172.21.15.90:51618) 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.280+0000 7f020aa00640 1 -- 172.21.15.90:0/3249024763 learned_addr learned my addr 172.21.15.90:0/3249024763 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020b400640 1 --2- 172.21.15.90:0/3249024763 >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 0x7f020c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020aa00640 1 -- 172.21.15.90:0/3249024763 >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 msgr2=0x7f020c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020aa00640 1 --2- 172.21.15.90:0/3249024763 >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 0x7f020c15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020aa00640 1 -- 172.21.15.90:0/3249024763 >> v1:172.21.15.102:6789/0 conn(0x7f020c06c7b0 legacy=0x7f020c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020aa00640 1 -- 172.21.15.90:0/3249024763 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c15f4a0 con 0x7f020c15e1b0 2024-03-31T15:55:48.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020b400640 1 --2- 172.21.15.90:0/3249024763 >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 0x7f020c15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:48.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020aa00640 1 --2- 172.21.15.90:0/3249024763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c15e1b0 0x7f020c162740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f020c06c5b0 tx=0x7f01fc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bebf3d91038c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020a000640 1 -- 172.21.15.90:0/3249024763 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0ab620 con 0x7f020c15e1b0 2024-03-31T15:55:48.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020a000640 1 -- 172.21.15.90:0/3249024763 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01fc0b0070 con 0x7f020c15e1b0 2024-03-31T15:55:48.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f020a000640 1 -- 172.21.15.90:0/3249024763 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0ab9e0 con 0x7f020c15e1b0 2024-03-31T15:55:48.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f0211c00640 1 -- 172.21.15.90:0/3249024763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c15e1b0 msgr2=0x7f020c162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f0211c00640 1 --2- 172.21.15.90:0/3249024763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c15e1b0 0x7f020c162740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f020c06c5b0 tx=0x7f01fc01c420 comp rx=0 tx=0).stop 2024-03-31T15:55:48.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f0211c00640 1 -- 172.21.15.90:0/3249024763 shutdown_connections 2024-03-31T15:55:48.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f0211c00640 1 --2- 172.21.15.90:0/3249024763 >> v2:172.21.15.102:3300/0 conn(0x7f020c06d7c0 0x7f020c15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.284+0000 7f0211c00640 1 --2- 172.21.15.90:0/3249024763 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c15e1b0 0x7f020c162740 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- 172.21.15.90:0/3249024763 >> 172.21.15.90:0/3249024763 conn(0x7f020c09aa40 msgr2=0x7f020c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- 172.21.15.90:0/3249024763 shutdown_connections 2024-03-31T15:55:48.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- 172.21.15.90:0/3249024763 wait complete. 2024-03-31T15:55:48.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 Processor -- start 2024-03-31T15:55:48.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- start start 2024-03-31T15:55:48.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f020c06d7c0 0x7f020c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f020c15e1b0 0x7f020c0b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f020c0d2940 con 0x7f020c06c7b0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f020c0d27c0 con 0x7f020c15e1b0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0211c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f020c0d2ac0 con 0x7f020c06d7c0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51624/0 (socket says 172.21.15.90:51624) 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 -- 172.21.15.90:0/762455119 learned_addr learned my addr 172.21.15.90:0/762455119 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020b400640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f020c15e1b0 0x7f020c0b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0203e00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f020c06d7c0 0x7f020c152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 -- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f020c06d7c0 msgr2=0x7f020c152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f020c06d7c0 0x7f020c152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 -- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f020c15e1b0 msgr2=0x7f020c0b78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f020c15e1b0 0x7f020c0b78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 -- 172.21.15.90:0/762455119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c0b7df0 con 0x7f020c06c7b0 2024-03-31T15:55:48.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f020aa00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f01fc01a050 tx=0x7f01fc01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.288+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0abd70 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01fc0be3f0 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01fc0be7e0 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020c0cb180 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020c0b4f50 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f020c0686d0 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f01fc0c7440 con 0x7f020c06c7b0 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0202a00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 0x7f01e806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f01fc1211b0 con 0x7f020c06c7b0 2024-03-31T15:55:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0203e00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 0x7f01e806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.292+0000 7f0203e00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 0x7f01e806a870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f020c0cb0b0 tx=0x7f01f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.296+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01fc0160d0 con 0x7f020c06c7b0 2024-03-31T15:55:48.450 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.456+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f020c0b1b60 con 0x7f020c06c7b0 2024-03-31T15:55:48.451 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.456+0000 7f0202a00640 1 -- 172.21.15.90:0/762455119 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f01fc0f1430 con 0x7f020c06c7b0 2024-03-31T15:55:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:48.452 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:48.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 msgr2=0x7f01e806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 0x7f01e806a870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f020c0cb0b0 tx=0x7f01f4008040 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 msgr2=0x7f020c0cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f01fc01a050 tx=0x7f01fc01ac60 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 shutdown_connections 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f01e80683b0 0x7f01e806a870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f020c15e1b0 0x7f020c0b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f020c06d7c0 0x7f020c152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 --2- 172.21.15.90:0/762455119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f020c06c7b0 0x7f020c0cab70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 >> 172.21.15.90:0/762455119 conn(0x7f020c09aa40 msgr2=0x7f020c15f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 shutdown_connections 2024-03-31T15:55:48.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.460+0000 7f0211c00640 1 -- 172.21.15.90:0/762455119 wait complete. 2024-03-31T15:55:48.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:48.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:48.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 Processor -- start 2024-03-31T15:55:48.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 -- start start 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1340ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb13406d460 0x7fb1340cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb13415e8c0 con 0x7fb13415e4e0 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fb1340d0ff0 con 0x7fb13406d460 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb138a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb1340d12d0 con 0x7fb13406d020 2024-03-31T15:55:48.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb132800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fb13406d460 0x7fb1340cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb131e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb13415e4e0 0x7fb13406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56758/0 (socket says 172.21.15.90:56758) 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb131e00640 1 -- 172.21.15.90:0/1146360161 learned_addr learned my addr 172.21.15.90:0/1146360161 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb12be00640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1340ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb132800640 1 -- 172.21.15.90:0/1146360161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 msgr2=0x7fb1340ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb132800640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1340ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb132800640 1 -- 172.21.15.90:0/1146360161 >> v1:172.21.15.90:6789/0 conn(0x7fb13415e4e0 legacy=0x7fb13406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.620+0000 7fb132800640 1 -- 172.21.15.90:0/1146360161 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1340cc710 con 0x7fb13406d460 2024-03-31T15:55:48.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb132800640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13406d460 0x7fb1340cd450 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009d90 tx=0x7fb12c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179fae0b8a388bb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb131400640 1 -- 172.21.15.90:0/1146360161 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0b2070 con 0x7fb13406d460 2024-03-31T15:55:48.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb131400640 1 -- 172.21.15.90:0/1146360161 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12c0a57a0 con 0x7fb13406d460 2024-03-31T15:55:48.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb131400640 1 -- 172.21.15.90:0/1146360161 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0a5b70 con 0x7fb13406d460 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 -- 172.21.15.90:0/1146360161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13406d460 msgr2=0x7fb1340cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13406d460 0x7fb1340cd450 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009d90 tx=0x7fb12c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 -- 172.21.15.90:0/1146360161 shutdown_connections 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13406d460 0x7fb1340cd450 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 --2- 172.21.15.90:0/1146360161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1340ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 -- 172.21.15.90:0/1146360161 >> 172.21.15.90:0/1146360161 conn(0x7fb1340d1f20 msgr2=0x7fb1340d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 -- 172.21.15.90:0/1146360161 shutdown_connections 2024-03-31T15:55:48.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.624+0000 7fb138a00640 1 -- 172.21.15.90:0/1146360161 wait complete. 2024-03-31T15:55:48.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 Processor -- start 2024-03-31T15:55:48.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 -- start start 2024-03-31T15:55:48.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1341597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb13406d460 0x7fb1340cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb1340d0d90 con 0x7fb13415e4e0 2024-03-31T15:55:48.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb1340d0c10 con 0x7fb13406d460 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb138a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb13416b030 con 0x7fb13406d020 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51632/0 (socket says 172.21.15.90:51632) 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 -- 172.21.15.90:0/55681157 learned_addr learned my addr 172.21.15.90:0/55681157 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb12be00640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb13406d460 0x7fb1340cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb131e00640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1341597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 -- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 msgr2=0x7fb1341597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1341597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 -- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb13406d460 msgr2=0x7fb1340cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb13406d460 0x7fb1340cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 -- 172.21.15.90:0/55681157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb134158720 con 0x7fb13415e4e0 2024-03-31T15:55:48.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb131e00640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1341597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.628+0000 7fb132800640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0b0040 tx=0x7fb12c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0b2050 con 0x7fb13415e4e0 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12c0be070 con 0x7fb13415e4e0 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb138a00640 1 -- 172.21.15.90:0/55681157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb134158a60 con 0x7fb13415e4e0 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0ada90 con 0x7fb13415e4e0 2024-03-31T15:55:48.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb138a00640 1 -- 172.21.15.90:0/55681157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1341592a0 con 0x7fb13415e4e0 2024-03-31T15:55:48.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1040052a0 con 0x7fb13415e4e0 2024-03-31T15:55:48.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb12c0adc30 con 0x7fb13415e4e0 2024-03-31T15:55:48.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 0x7fb11006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.632+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fb12c121da0 con 0x7fb13415e4e0 2024-03-31T15:55:48.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.636+0000 7fb131e00640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 0x7fb11006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.636+0000 7fb131e00640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 0x7fb11006a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb124000f70 tx=0x7fb124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.636+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb134158a60 con 0x7fb13415e4e0 2024-03-31T15:55:48.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb104005da0 con 0x7fb13415e4e0 2024-03-31T15:55:48.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb12a000640 1 -- 172.21.15.90:0/55681157 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7fb12c016560 con 0x7fb13415e4e0 2024-03-31T15:55:48.790 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 msgr2=0x7fb11006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 0x7fb11006a840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb124000f70 tx=0x7fb124009040 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 msgr2=0x7fb134070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0b0040 tx=0x7fb12c019680 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 shutdown_connections 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb110068380 0x7fb11006a840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb13415e4e0 0x7fb134070830 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb13406d460 0x7fb1340cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 --2- 172.21.15.90:0/55681157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb13406d020 0x7fb1341597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 >> 172.21.15.90:0/55681157 conn(0x7fb1340d1f20 msgr2=0x7fb1340c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 shutdown_connections 2024-03-31T15:55:48.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.796+0000 7fb123400640 1 -- 172.21.15.90:0/55681157 wait complete. 2024-03-31T15:55:48.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:48.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.956+0000 7fd86bc00640 1 Processor -- start 2024-03-31T15:55:48.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.956+0000 7fd86bc00640 1 -- start start 2024-03-31T15:55:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd86bc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd86415e360 0x7fd86415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd86bc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd86415e740 0x7fd864071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd86bc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8640731e0 con 0x7fd86406e710 2024-03-31T15:55:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd86bc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd864072660 con 0x7fd86415e360 2024-03-31T15:55:48.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd86bc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd864072940 con 0x7fd86415e740 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd86415e360 0x7fd86415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd86415e360 0x7fd86415f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51638/0 (socket says 172.21.15.90:51638) 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 -- 172.21.15.90:0/407954897 learned_addr learned my addr 172.21.15.90:0/407954897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd863e00640 1 --2- 172.21.15.90:0/407954897 >> v2:172.21.15.102:3300/0 conn(0x7fd86415e740 0x7fd864071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 -- 172.21.15.90:0/407954897 >> v2:172.21.15.102:3300/0 conn(0x7fd86415e740 msgr2=0x7fd864071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 --2- 172.21.15.90:0/407954897 >> v2:172.21.15.102:3300/0 conn(0x7fd86415e740 0x7fd864071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 -- 172.21.15.90:0/407954897 >> v1:172.21.15.102:6789/0 conn(0x7fd86406e710 legacy=0x7fd8640ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:48.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.960+0000 7fd869200640 1 -- 172.21.15.90:0/407954897 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd864072c20 con 0x7fd86415e360 2024-03-31T15:55:48.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd869200640 1 --2- 172.21.15.90:0/407954897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86415e360 0x7fd86415f7a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd858009870 tx=0x7fd85801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f487a8501a5b6416 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd863400640 1 -- 172.21.15.90:0/407954897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580b1070 con 0x7fd86415e360 2024-03-31T15:55:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd863400640 1 -- 172.21.15.90:0/407954897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8580ac410 con 0x7fd86415e360 2024-03-31T15:55:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd863400640 1 -- 172.21.15.90:0/407954897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580ac7e0 con 0x7fd86415e360 2024-03-31T15:55:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- 172.21.15.90:0/407954897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86415e360 msgr2=0x7fd86415f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 --2- 172.21.15.90:0/407954897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86415e360 0x7fd86415f7a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd858009870 tx=0x7fd85801c420 comp rx=0 tx=0).stop 2024-03-31T15:55:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- 172.21.15.90:0/407954897 shutdown_connections 2024-03-31T15:55:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 --2- 172.21.15.90:0/407954897 >> v2:172.21.15.102:3300/0 conn(0x7fd86415e740 0x7fd864071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 --2- 172.21.15.90:0/407954897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86415e360 0x7fd86415f7a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- 172.21.15.90:0/407954897 >> 172.21.15.90:0/407954897 conn(0x7fd8640b1c50 msgr2=0x7fd8640b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- 172.21.15.90:0/407954897 shutdown_connections 2024-03-31T15:55:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- 172.21.15.90:0/407954897 wait complete. 2024-03-31T15:55:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 Processor -- start 2024-03-31T15:55:48.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.964+0000 7fd86bc00640 1 -- start start 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86415e360 0x7fd86409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86415e740 0x7fd8640c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd864168150 con 0x7fd86406e710 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd864167fd0 con 0x7fd86415e740 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd8641682d0 con 0x7fd86415e360 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51640/0 (socket says 172.21.15.90:51640) 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 -- 172.21.15.90:0/1215169536 learned_addr learned my addr 172.21.15.90:0/1215169536 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd863e00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86415e360 0x7fd86409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869c00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86415e740 0x7fd8640c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 -- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86415e360 msgr2=0x7fd86409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86415e360 0x7fd86409aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 -- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86415e740 msgr2=0x7fd8640c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:48.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86415e740 0x7fd8640c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:48.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 -- 172.21.15.90:0/1215169536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8640b5390 con 0x7fd86406e710 2024-03-31T15:55:48.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd869200640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd8580af040 tx=0x7fd8580191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580bc070 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8580ace40 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580b1070 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8640b50c0 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8640c85c0 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8640685f0 con 0x7fd86406e710 2024-03-31T15:55:48.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.968+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8580bf070 con 0x7fd86406e710 2024-03-31T15:55:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.972+0000 7fd860c00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 0x7fd84406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.972+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fd858120530 con 0x7fd86406e710 2024-03-31T15:55:48.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.972+0000 7fd863e00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 0x7fd84406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:48.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.972+0000 7fd863e00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 0x7fd84406a840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd8640cfb90 tx=0x7fd850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:48.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:48.976+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd8580160d0 con 0x7fd86406e710 2024-03-31T15:55:49.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.144+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd8640ce250 con 0x7fd86406e710 2024-03-31T15:55:49.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.144+0000 7fd860c00640 1 -- 172.21.15.90:0/1215169536 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7fd8580f07b0 con 0x7fd86406e710 2024-03-31T15:55:49.138 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 msgr2=0x7fd84406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 0x7fd84406a840 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd8640cfb90 tx=0x7fd850008040 comp rx=0 tx=0).stop 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 msgr2=0x7fd8640b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fd8580af040 tx=0x7fd8580191c0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 shutdown_connections 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd844068380 0x7fd84406a840 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86415e740 0x7fd8640c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd86415e360 0x7fd86409aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 --2- 172.21.15.90:0/1215169536 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd86406e710 0x7fd8640b4b80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 >> 172.21.15.90:0/1215169536 conn(0x7fd8640b1c50 msgr2=0x7fd8640ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 shutdown_connections 2024-03-31T15:55:49.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.148+0000 7fd86bc00640 1 -- 172.21.15.90:0/1215169536 wait complete. 2024-03-31T15:55:49.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.304+0000 7f1d55e00640 1 Processor -- start 2024-03-31T15:55:49.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.304+0000 7f1d55e00640 1 -- start start 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d5015bed0 0x7f1d5006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5006f2f0 0x7f1d5006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d500cefb0 con 0x7f1d5006e3e0 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1d500cf290 con 0x7f1d5015bed0 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d500cf570 con 0x7f1d5006f2f0 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1d5006e3e0 0x7f1d5015fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56768/0 (socket says 172.21.15.90:56768) 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d55400640 1 -- 172.21.15.90:0/1427281583 learned_addr learned my addr 172.21.15.90:0/1427281583 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d54a00640 1 --2- 172.21.15.90:0/1427281583 >> v2:172.21.15.102:3300/0 conn(0x7f1d5015bed0 0x7f1d5006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d47e00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5006f2f0 0x7f1d5006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d4fe00640 1 -- 172.21.15.90:0/1427281583 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633959397 0 0) 0x7f1d500cefb0 con 0x7f1d5006e3e0 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d4fe00640 1 -- 172.21.15.90:0/1427281583 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d34003530 con 0x7f1d5006e3e0 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d54a00640 1 -- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5006f2f0 msgr2=0x7f1d5006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d54a00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5006f2f0 0x7f1d5006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d54a00640 1 -- 172.21.15.90:0/1427281583 >> v1:172.21.15.90:6789/0 conn(0x7f1d5006e3e0 legacy=0x7f1d5015fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.308+0000 7f1d54a00640 1 -- 172.21.15.90:0/1427281583 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d500cf850 con 0x7f1d5015bed0 2024-03-31T15:55:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d54a00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5015bed0 0x7f1d5006d960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d400097c0 tx=0x7f1d4001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c7d87a378b44bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d4fe00640 1 -- 172.21.15.90:0/1427281583 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d400b0070 con 0x7f1d5015bed0 2024-03-31T15:55:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d4fe00640 1 -- 172.21.15.90:0/1427281583 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d400ab600 con 0x7f1d5015bed0 2024-03-31T15:55:49.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d4fe00640 1 -- 172.21.15.90:0/1427281583 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d400ab9b0 con 0x7f1d5015bed0 2024-03-31T15:55:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5015bed0 msgr2=0x7f1d5006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5015bed0 0x7f1d5006d960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1d400097c0 tx=0x7f1d4001c470 comp rx=0 tx=0).stop 2024-03-31T15:55:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- 172.21.15.90:0/1427281583 shutdown_connections 2024-03-31T15:55:49.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5006f2f0 0x7f1d5006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 --2- 172.21.15.90:0/1427281583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5015bed0 0x7f1d5006d960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- 172.21.15.90:0/1427281583 >> 172.21.15.90:0/1427281583 conn(0x7f1d500d0390 msgr2=0x7f1d500d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- 172.21.15.90:0/1427281583 shutdown_connections 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- 172.21.15.90:0/1427281583 wait complete. 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 Processor -- start 2024-03-31T15:55:49.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.312+0000 7f1d55e00640 1 -- start start 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5006f2f0 0x7f1d50159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5015bed0 0x7f1d50159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5015c430 con 0x7f1d5006e3e0 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5015c2b0 con 0x7f1d5006f2f0 2024-03-31T15:55:49.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1d500ceb60 con 0x7f1d5015bed0 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51656/0 (socket says 172.21.15.90:51656) 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 -- 172.21.15.90:0/110965263 learned_addr learned my addr 172.21.15.90:0/110965263 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55400640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5015bed0 0x7f1d50159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d47e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5006f2f0 0x7f1d50159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 -- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5015bed0 msgr2=0x7f1d50159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5015bed0 0x7f1d50159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 -- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5006f2f0 msgr2=0x7f1d50159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5006f2f0 0x7f1d50159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 -- 172.21.15.90:0/110965263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d501f68e0 con 0x7f1d5006e3e0 2024-03-31T15:55:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d54a00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1d400ae040 tx=0x7f1d40019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d400b0040 con 0x7f1d5006e3e0 2024-03-31T15:55:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d400ab400 con 0x7f1d5006e3e0 2024-03-31T15:55:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d50159c60 con 0x7f1d5006e3e0 2024-03-31T15:55:49.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d400ab9b0 con 0x7f1d5006e3e0 2024-03-31T15:55:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d501f6b80 con 0x7f1d5006e3e0 2024-03-31T15:55:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.316+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d50068d00 con 0x7f1d5006e3e0 2024-03-31T15:55:49.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.320+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d400bd020 con 0x7f1d5006e3e0 2024-03-31T15:55:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.320+0000 7f1d4d600640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 0x7f1d2c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.320+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f1d40120560 con 0x7f1d5006e3e0 2024-03-31T15:55:49.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.320+0000 7f1d47e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 0x7f1d2c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.320+0000 7f1d47e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 0x7f1d2c06a840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1d501590c0 tx=0x7f1d38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.324+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1d40016090 con 0x7f1d5006e3e0 2024-03-31T15:55:49.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1d5014fbd0 con 0x7f1d5006e3e0 2024-03-31T15:55:49.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d4d600640 1 -- 172.21.15.90:0/110965263 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f1d400f0100 con 0x7f1d5006e3e0 2024-03-31T15:55:49.473 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 msgr2=0x7f1d2c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 0x7f1d2c06a840 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1d501590c0 tx=0x7f1d38009190 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 msgr2=0x7f1d5014f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1d400ae040 tx=0x7f1d40019210 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 shutdown_connections 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1d2c068380 0x7f1d2c06a840 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1d5015bed0 0x7f1d50159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d5006f2f0 0x7f1d50159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 --2- 172.21.15.90:0/110965263 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1d5006e3e0 0x7f1d5014f690 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 >> 172.21.15.90:0/110965263 conn(0x7f1d500d0390 msgr2=0x7f1d500d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 shutdown_connections 2024-03-31T15:55:49.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.480+0000 7f1d55e00640 1 -- 172.21.15.90:0/110965263 wait complete. 2024-03-31T15:55:49.493 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:55:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.640+0000 7f252ac00640 1 Processor -- start 2024-03-31T15:55:49.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.640+0000 7f252ac00640 1 -- start start 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f252ac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 0x7f25240b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f252ac00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252406c2d0 0x7f2524072d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f252ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25240b3f10 con 0x7f252406b5f0 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f252ac00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f25240b3cc0 con 0x7f252406c2d0 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f252ac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2524072880 con 0x7f252415e3b0 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252406c2d0 0x7f2524072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f252406c2d0 0x7f2524072d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51664/0 (socket says 172.21.15.90:51664) 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 -- 172.21.15.90:0/3857150930 learned_addr learned my addr 172.21.15.90:0/3857150930 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2523400640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 0x7f25240b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 -- 172.21.15.90:0/3857150930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 msgr2=0x7f25240b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 0x7f25240b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 -- 172.21.15.90:0/3857150930 >> v1:172.21.15.102:6789/0 conn(0x7f252406b5f0 legacy=0x7f252406b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2528c00640 1 -- 172.21.15.90:0/3857150930 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25240b41f0 con 0x7f252406c2d0 2024-03-31T15:55:49.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.644+0000 7f2523400640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 0x7f25240b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:55:49.640 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f2528c00640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f2524072d10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f251c009e50 tx=0x7f251c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bbe98f9cd9be20 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f2522a00640 1 -- 172.21.15.90:0/3857150930 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0a5860 con 0x7f252406c2d0 2024-03-31T15:55:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f2522a00640 1 -- 172.21.15.90:0/3857150930 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c0a5a00 con 0x7f252406c2d0 2024-03-31T15:55:49.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f2522a00640 1 -- 172.21.15.90:0/3857150930 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0a5dd0 con 0x7f252406c2d0 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- 172.21.15.90:0/3857150930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 msgr2=0x7f2524072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f2524072d10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f251c009e50 tx=0x7f251c01c9c0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- 172.21.15.90:0/3857150930 shutdown_connections 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f2524072d10 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 --2- 172.21.15.90:0/3857150930 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252415e3b0 0x7f25240b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- 172.21.15.90:0/3857150930 >> 172.21.15.90:0/3857150930 conn(0x7f25240b29c0 msgr2=0x7f25240b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- 172.21.15.90:0/3857150930 shutdown_connections 2024-03-31T15:55:49.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- 172.21.15.90:0/3857150930 wait complete. 2024-03-31T15:55:49.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 Processor -- start 2024-03-31T15:55:49.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.648+0000 7f252ac00640 1 -- start start 2024-03-31T15:55:49.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252406b5f0 0x7f2524156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252415e3b0 0x7f252415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f252416b480 con 0x7f252406c2d0 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f252416b300 con 0x7f252415e3b0 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f252416b600 con 0x7f252406b5f0 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51680/0 (socket says 172.21.15.90:51680) 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 -- 172.21.15.90:0/4004353617 learned_addr learned my addr 172.21.15.90:0/4004353617 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2528c00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252415e3b0 0x7f252415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523e00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252406b5f0 0x7f2524156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 -- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252406b5f0 msgr2=0x7f2524156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252406b5f0 0x7f2524156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 -- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252415e3b0 msgr2=0x7f252415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252415e3b0 0x7f252415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 -- 172.21.15.90:0/4004353617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2524156f40 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f2523400640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f251000c7f0 tx=0x7f251000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510015070 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2510007820 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510007c10 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2524157220 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25241577e0 con 0x7f252406c2d0 2024-03-31T15:55:49.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2524075880 con 0x7f252406c2d0 2024-03-31T15:55:49.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.652+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2510018070 con 0x7f252406c2d0 2024-03-31T15:55:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.656+0000 7f251be00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 0x7f250406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.656+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f2510086d90 con 0x7f252406c2d0 2024-03-31T15:55:49.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.656+0000 7f2523e00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 0x7f250406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.656+0000 7f2523e00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 0x7f250406a840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2514007fd0 tx=0x7f2514007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.660+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f25100569d0 con 0x7f252406c2d0 2024-03-31T15:55:49.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f25240685f0 con 0x7f252406c2d0 2024-03-31T15:55:49.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f251be00640 1 -- 172.21.15.90:0/4004353617 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f251005acd0 con 0x7f252406c2d0 2024-03-31T15:55:49.817 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 msgr2=0x7f250406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 0x7f250406a840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2514007fd0 tx=0x7f2514007e40 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 msgr2=0x7f252414b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f251000c7f0 tx=0x7f251000ccb0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 shutdown_connections 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2504068380 0x7f250406a840 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f252415e3b0 0x7f252415f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f252406c2d0 0x7f252414b4b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 --2- 172.21.15.90:0/4004353617 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f252406b5f0 0x7f2524156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 >> 172.21.15.90:0/4004353617 conn(0x7f25240b29c0 msgr2=0x7f25240afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 shutdown_connections 2024-03-31T15:55:49.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.824+0000 7f252ac00640 1 -- 172.21.15.90:0/4004353617 wait complete. 2024-03-31T15:55:49.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:55:49.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 Processor -- start 2024-03-31T15:55:49.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 -- start start 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f65a815e320 0x7f65a80d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65a80d2940 0x7f65a8161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a80d2df0 con 0x7f65a81622d0 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f65a815e700 con 0x7f65a815e320 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65ae800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f65a8162720 con 0x7f65a80d2940 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f65a815e320 0x7f65a80d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f65a81622d0 0x7f65a806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:56780/0 (socket says 172.21.15.90:56780) 2024-03-31T15:55:49.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7e00640 1 -- 172.21.15.90:0/274534798 learned_addr learned my addr 172.21.15.90:0/274534798 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a6a00640 1 --2- 172.21.15.90:0/274534798 >> v2:172.21.15.102:3300/0 conn(0x7f65a80d2940 0x7f65a8161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7400640 1 -- 172.21.15.90:0/274534798 >> v2:172.21.15.102:3300/0 conn(0x7f65a80d2940 msgr2=0x7f65a8161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7400640 1 --2- 172.21.15.90:0/274534798 >> v2:172.21.15.102:3300/0 conn(0x7f65a80d2940 0x7f65a8161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7400640 1 -- 172.21.15.90:0/274534798 >> v1:172.21.15.90:6789/0 conn(0x7f65a81622d0 legacy=0x7f65a806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a7400640 1 -- 172.21.15.90:0/274534798 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a80d1400 con 0x7f65a815e320 2024-03-31T15:55:49.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.992+0000 7f65a6000640 1 -- 172.21.15.90:0/274534798 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674276741 0 0) 0x7f65a80d2df0 con 0x7f65a81622d0 2024-03-31T15:55:49.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65a7400640 1 --2- 172.21.15.90:0/274534798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a815e320 0x7f65a80d0d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f65980097f0 tx=0x7f659801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacedc1bda70c0e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65a6000640 1 -- 172.21.15.90:0/274534798 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65980ab680 con 0x7f65a815e320 2024-03-31T15:55:49.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65a6000640 1 -- 172.21.15.90:0/274534798 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65980b0070 con 0x7f65a815e320 2024-03-31T15:55:49.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65a6000640 1 -- 172.21.15.90:0/274534798 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65980aba40 con 0x7f65a815e320 2024-03-31T15:55:49.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 -- 172.21.15.90:0/274534798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a815e320 msgr2=0x7f65a80d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 --2- 172.21.15.90:0/274534798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a815e320 0x7f65a80d0d20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f65980097f0 tx=0x7f659801c460 comp rx=0 tx=0).stop 2024-03-31T15:55:49.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 -- 172.21.15.90:0/274534798 shutdown_connections 2024-03-31T15:55:49.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 --2- 172.21.15.90:0/274534798 >> v2:172.21.15.102:3300/0 conn(0x7f65a80d2940 0x7f65a8161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 --2- 172.21.15.90:0/274534798 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a815e320 0x7f65a80d0d20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 -- 172.21.15.90:0/274534798 >> 172.21.15.90:0/274534798 conn(0x7f65a808bfd0 msgr2=0x7f65a80d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 -- 172.21.15.90:0/274534798 shutdown_connections 2024-03-31T15:55:49.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:49.996+0000 7f65ae800640 1 -- 172.21.15.90:0/274534798 wait complete. 2024-03-31T15:55:49.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 Processor -- start 2024-03-31T15:55:49.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- start start 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a815e320 0x7f65a80c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 0x7f65a8143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65a816b080 con 0x7f65a80d2940 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65a816af00 con 0x7f65a815e320 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f65a816b200 con 0x7f65a81622d0 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.995 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51694/0 (socket says 172.21.15.90:51694) 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 -- 172.21.15.90:0/3642326537 learned_addr learned my addr 172.21.15.90:0/3642326537 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a6a00640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a815e320 0x7f65a80c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7e00640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 0x7f65a8143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 -- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 msgr2=0x7f65a8143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 0x7f65a8143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 -- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a815e320 msgr2=0x7f65a80c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a815e320 0x7f65a80c7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 -- 172.21.15.90:0/3642326537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a80c88c0 con 0x7f65a80d2940 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7e00640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 0x7f65a8143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65a7400640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65980ad040 tx=0x7f659801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598006e20 con 0x7f65a80d2940 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65980a7070 con 0x7f65a80d2940 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6598004420 con 0x7f65a80d2940 2024-03-31T15:55:49.996 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65a80c59f0 con 0x7f65a80d2940 2024-03-31T15:55:49.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.000+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65a80c80c0 con 0x7f65a80d2940 2024-03-31T15:55:49.997 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.004+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65a8068660 con 0x7f65a80d2940 2024-03-31T15:55:49.998 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.004+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6598003620 con 0x7f65a80d2940 2024-03-31T15:55:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.004+0000 7f659f400640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 0x7f658806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.004+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f65981214d0 con 0x7f65a80d2940 2024-03-31T15:55:49.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.004+0000 7f65a6a00640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 0x7f658806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:50.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.008+0000 7f65a6a00640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 0x7f658806a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65a815d080 tx=0x7f6594008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:50.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.008+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6598016050 con 0x7f65a80d2940 2024-03-31T15:55:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.184+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f65a8157a50 con 0x7f65a80d2940 2024-03-31T15:55:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.184+0000 7f659f400640 1 -- 172.21.15.90:0/3642326537 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f65980f1190 con 0x7f65a80d2940 2024-03-31T15:55:50.177 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:55:50.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 msgr2=0x7f658806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:50.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 0x7f658806a870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f65a815d080 tx=0x7f6594008040 comp rx=0 tx=0).stop 2024-03-31T15:55:50.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 msgr2=0x7f65a80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:50.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65980ad040 tx=0x7f659801aca0 comp rx=0 tx=0).stop 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 shutdown_connections 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65880683b0 0x7f658806a870 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65a81622d0 0x7f65a8143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a815e320 0x7f65a80c7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 --2- 172.21.15.90:0/3642326537 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65a80d2940 0x7f65a80cbc20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 >> 172.21.15.90:0/3642326537 conn(0x7f65a808bfd0 msgr2=0x7f65a80af050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 shutdown_connections 2024-03-31T15:55:50.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:50.188+0000 7f65ae800640 1 -- 172.21.15.90:0/3642326537 wait complete. 2024-03-31T15:55:50.200 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:55:50.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:50.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:50.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39400]: fuse finished with error 0 and tester_r 0 2024-03-31T15:55:50.222 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:55:56.324 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:55:56.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:56.324 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:55:56.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-31T15:55:56.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 Processor -- start 2024-03-31T15:55:56.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 -- start start 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe140ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1415e8c0 con 0x7efe1406d460 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe140d0ff0 con 0x7efe1415e4e0 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.524+0000 7efe19200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7efe140d12d0 con 0x7efe1406d020 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe12800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7efe1406d460 0x7efe140cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45136/0 (socket says 172.21.15.90:45136) 2024-03-31T15:55:56.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe12800640 1 -- 172.21.15.90:0/1599944460 learned_addr learned my addr 172.21.15.90:0/1599944460 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe11400640 1 --2- 172.21.15.90:0/1599944460 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe140ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590038781 0 0) 0x7efe1415e8c0 con 0x7efe1406d460 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdf8003530 con 0x7efe1406d460 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481217811 0 0) 0x7efe140d0ff0 con 0x7efe1415e4e0 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe1415e8c0 con 0x7efe1415e4e0 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe11400640 1 -- 172.21.15.90:0/1599944460 >> v1:172.21.15.102:6789/0 conn(0x7efe1415e4e0 legacy=0x7efe1406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe11400640 1 -- 172.21.15.90:0/1599944460 >> v1:172.21.15.90:6789/0 conn(0x7efe1406d460 legacy=0x7efe140cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.528+0000 7efe11400640 1 -- 172.21.15.90:0/1599944460 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe140cc710 con 0x7efe1406d020 2024-03-31T15:55:56.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe11400640 1 --2- 172.21.15.90:0/1599944460 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe140ccf10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efe000097f0 tx=0x7efe0001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a8baee686e1d531 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000b0070 con 0x7efe1406d020 2024-03-31T15:55:56.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe000ab600 con 0x7efe1406d020 2024-03-31T15:55:56.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe10a00640 1 -- 172.21.15.90:0/1599944460 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000ab9b0 con 0x7efe1406d020 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- 172.21.15.90:0/1599944460 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 msgr2=0x7efe140ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 --2- 172.21.15.90:0/1599944460 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe140ccf10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efe000097f0 tx=0x7efe0001c450 comp rx=0 tx=0).stop 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- 172.21.15.90:0/1599944460 shutdown_connections 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 --2- 172.21.15.90:0/1599944460 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe140ccf10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- 172.21.15.90:0/1599944460 >> 172.21.15.90:0/1599944460 conn(0x7efe140d1f20 msgr2=0x7efe140d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- 172.21.15.90:0/1599944460 shutdown_connections 2024-03-31T15:55:56.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- 172.21.15.90:0/1599944460 wait complete. 2024-03-31T15:55:56.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 Processor -- start 2024-03-31T15:55:56.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- start start 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe1415a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1415e4e0 0x7efe14152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efe140d0d90 con 0x7efe1406d460 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe140d0c10 con 0x7efe1415e4e0 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe19200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7efe1416c040 con 0x7efe1406d020 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe11400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe11400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45766/0 (socket says 172.21.15.90:45766) 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.532+0000 7efe11400640 1 -- 172.21.15.90:0/4038987047 learned_addr learned my addr 172.21.15.90:0/4038987047 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11e00640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe1415a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe12800640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1415e4e0 0x7efe14152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 -- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 msgr2=0x7efe1415a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe1415a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 -- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1415e4e0 msgr2=0x7efe14152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1415e4e0 0x7efe14152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 -- 172.21.15.90:0/4038987047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1415abd0 con 0x7efe1406d460 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11e00640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe1415a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe11400640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efe000097f0 tx=0x7efe140ce370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000b8020 con 0x7efe1406d460 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe000b4410 con 0x7efe1406d460 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1415cf60 con 0x7efe1406d460 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe000b0040 con 0x7efe1406d460 2024-03-31T15:55:56.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1415d520 con 0x7efe1406d460 2024-03-31T15:55:56.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe140685f0 con 0x7efe1406d460 2024-03-31T15:55:56.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe000bb020 con 0x7efe1406d460 2024-03-31T15:55:56.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 0x7efdf406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.536+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7efe00120360 con 0x7efe1406d460 2024-03-31T15:55:56.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.540+0000 7efe11e00640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 0x7efdf406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.540+0000 7efe11e00640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 0x7efdf406a870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe140b7e10 tx=0x7efe04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.544+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7efe000c2070 con 0x7efe1406d460 2024-03-31T15:55:56.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.764+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7efe140c67e0 con 0x7efe1406d460 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.764+0000 7efe0a000640 1 -- 172.21.15.90:0/4038987047 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v15) ==== 211+0+168 (secure 0 0 0) 0x7efe000160d0 con 0x7efe1406d460 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow rwp" 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:55:56.759 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-31T15:55:56.760 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:55:56.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 msgr2=0x7efdf406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 0x7efdf406a870 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7efe140b7e10 tx=0x7efe04009040 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 msgr2=0x7efe14070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efe000097f0 tx=0x7efe140ce370 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 shutdown_connections 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efdf40683b0 0x7efdf406a870 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1415e4e0 0x7efe14152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe1406d460 0x7efe14070290 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 --2- 172.21.15.90:0/4038987047 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7efe1406d020 0x7efe1415a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 >> 172.21.15.90:0/4038987047 conn(0x7efe140d1f20 msgr2=0x7efe14159450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 shutdown_connections 2024-03-31T15:55:56.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.768+0000 7efe19200640 1 -- 172.21.15.90:0/4038987047 wait complete. 2024-03-31T15:55:56.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:55:56.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 Processor -- start 2024-03-31T15:55:56.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 -- start start 2024-03-31T15:55:56.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d08e0 0x7fcadc06f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcadc160dd0 0x7fcadc162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcadc06fac0 con 0x7fcadc0d0cc0 2024-03-31T15:55:56.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcadc06fda0 con 0x7fcadc160dd0 2024-03-31T15:55:56.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadbe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcadc070080 con 0x7fcadc0d08e0 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadb400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcadc160dd0 0x7fcadc162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcad3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcadc0d0cc0 0x7fcadc161d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45148/0 (socket says 172.21.15.90:45148) 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcad3e00640 1 -- 172.21.15.90:0/1410649606 learned_addr learned my addr 172.21.15.90:0/1410649606 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadaa00640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d08e0 0x7fcadc06f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadb400640 1 -- 172.21.15.90:0/1410649606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d08e0 msgr2=0x7fcadc06f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadb400640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d08e0 0x7fcadc06f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadb400640 1 -- 172.21.15.90:0/1410649606 >> v1:172.21.15.90:6789/0 conn(0x7fcadc0d0cc0 legacy=0x7fcadc161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.924+0000 7fcadb400640 1 -- 172.21.15.90:0/1410649606 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc070360 con 0x7fcadc160dd0 2024-03-31T15:55:56.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadb400640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc162430 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009d90 tx=0x7fcad401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7196f82116a0a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcada000640 1 -- 172.21.15.90:0/1410649606 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40b2070 con 0x7fcadc160dd0 2024-03-31T15:55:56.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcada000640 1 -- 172.21.15.90:0/1410649606 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad40a5710 con 0x7fcadc160dd0 2024-03-31T15:55:56.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcada000640 1 -- 172.21.15.90:0/1410649606 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40a5ae0 con 0x7fcadc160dd0 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 -- 172.21.15.90:0/1410649606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 msgr2=0x7fcadc162430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc162430 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcad4009d90 tx=0x7fcad401c850 comp rx=0 tx=0).stop 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 -- 172.21.15.90:0/1410649606 shutdown_connections 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc162430 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 --2- 172.21.15.90:0/1410649606 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d08e0 0x7fcadc06f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 -- 172.21.15.90:0/1410649606 >> 172.21.15.90:0/1410649606 conn(0x7fcadc0d1d70 msgr2=0x7fcadc0d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 -- 172.21.15.90:0/1410649606 shutdown_connections 2024-03-31T15:55:56.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.928+0000 7fcadbe00640 1 -- 172.21.15.90:0/1410649606 wait complete. 2024-03-31T15:55:56.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 Processor -- start 2024-03-31T15:55:56.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- start start 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 0x7fcadc14cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d0cc0 0x7fcadc1f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc15e580 con 0x7fcadc160dd0 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc15e400 con 0x7fcadc0d08e0 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcadc15e700 con 0x7fcadc0d0cc0 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45780/0 (socket says 172.21.15.90:45780) 2024-03-31T15:55:56.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 -- 172.21.15.90:0/3772461816 learned_addr learned my addr 172.21.15.90:0/3772461816 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcad3e00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d0cc0 0x7fcadc1f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadaa00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 0x7fcadc14cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 -- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d0cc0 msgr2=0x7fcadc1f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d0cc0 0x7fcadc1f7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 -- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 msgr2=0x7fcadc14cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 0x7fcadc14cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 -- 172.21.15.90:0/3772461816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc156ef0 con 0x7fcadc160dd0 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadaa00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 0x7fcadc14cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:56.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadb400640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcad40b0040 tx=0x7fcad40195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40b2050 con 0x7fcadc160dd0 2024-03-31T15:55:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad40be070 con 0x7fcadc160dd0 2024-03-31T15:55:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40ad890 con 0x7fcadc160dd0 2024-03-31T15:55:56.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcadc1facc0 con 0x7fcadc160dd0 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.932+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcadc1fb280 con 0x7fcadc160dd0 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcadc0685f0 con 0x7fcadc160dd0 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcad40ada30 con 0x7fcadc160dd0 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcad2a00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 0x7fcab806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fcad4121db0 con 0x7fcadc160dd0 2024-03-31T15:55:56.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcadaa00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 0x7fcab806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:56.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.936+0000 7fcadaa00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 0x7fcab806a840 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007a70 tx=0x7fcacc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:56.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:56.940+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcad4126070 con 0x7fcadc160dd0 2024-03-31T15:55:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcadc0c8c00 con 0x7fcadc160dd0 2024-03-31T15:55:57.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcad2a00640 1 -- 172.21.15.90:0/3772461816 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+82 (secure 0 0 0) 0x7fcadc156ef0 con 0x7fcadc160dd0 2024-03-31T15:55:57.174 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-31T15:55:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 msgr2=0x7fcab806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 0x7fcab806a840 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007a70 tx=0x7fcacc008040 comp rx=0 tx=0).stop 2024-03-31T15:55:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 msgr2=0x7fcadc154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:57.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fcad40b0040 tx=0x7fcad40195f0 comp rx=0 tx=0).stop 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 shutdown_connections 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcab8068380 0x7fcab806a840 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcadc160dd0 0x7fcadc154470 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcadc0d0cc0 0x7fcadc1f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 --2- 172.21.15.90:0/3772461816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcadc0d08e0 0x7fcadc14cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 >> 172.21.15.90:0/3772461816 conn(0x7fcadc0d1d70 msgr2=0x7fcadc06c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 shutdown_connections 2024-03-31T15:55:57.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57.180+0000 7fcadbe00640 1 -- 172.21.15.90:0/3772461816 wait complete. 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:55:57.195 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:57.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.196 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:57.205 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42366sec preferred_lft 42366sec 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.206 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:57.207 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:55:57.208 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:55:57.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.208 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:55:57.269 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:55:57.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.271 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:55:57.326 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:55:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:55:57.328 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:55:57.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:55:57.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:57.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.461 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:55:57.470 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:55:57.470 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:55:57.470 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:57.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:55:57.535 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:55:57.535 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:57.535 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:57.587 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:55:57.588 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:55:57.588 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:57.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:57.701 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:57.758 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:57.759 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:57.760 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:57.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.761 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:57.813 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:55:57.813 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:55:57.863 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:55:57.885 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:55:57.938 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:55:57.939 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:57.940 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:55:57.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:57.941 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:55:57.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:55:57.964+0000 7fd11b279580 -1 init, newargv = 0x55aa3b4e6290 newargc=15 2024-03-31T15:55:57.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39856]: starting ceph client 2024-03-31T15:55:57.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39856]: starting fuse 2024-03-31T15:55:57.967 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:55:57.967 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:55:57.968 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:55:57.969 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:55:57.970 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:55:58.091 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.39856.asok 2024-03-31T15:55:58.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:55:58.103 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39856.asok 2024-03-31T15:55:58.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:58.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39856.asok status 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "39856", 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:58.271 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 5970, 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 5970 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 111726678 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:58.272 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 111726678 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.5970 192.168.0.1:0/111726678", 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/111726678", 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 18, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 43, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:55:58.273 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:55:58.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:58.286 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:55:58.286 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:58.295 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:55:58.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:58.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:58.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:55:58.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:58.354 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-31T15:55:58.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:55:58.407 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-31T15:55:58.461 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:55:58.462 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:55:58.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:55:58.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 Processor -- start 2024-03-31T15:55:58.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 -- start start 2024-03-31T15:55:58.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b1015bfd0 0x7f7b1006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b1006bd60 0x7f7b1015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b100b47e0 con 0x7f7b101449a0 2024-03-31T15:55:58.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1006c1d0 con 0x7f7b1015bfd0 2024-03-31T15:55:58.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b14c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b100b4ac0 con 0x7f7b1006bd60 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b1015bfd0 0x7f7b1006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0e800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b1006bd60 0x7f7b1015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b101449a0 0x7f7b100b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42716/0 (socket says 172.21.15.90:42716) 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b1015bfd0 0x7f7b1006b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:38278/0 (socket says 172.21.15.90:38278) 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0de00640 1 -- 172.21.15.90:0/2238547366 learned_addr learned my addr 172.21.15.90:0/2238547366 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 -- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b1006bd60 msgr2=0x7f7b1015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 --2- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b1006bd60 0x7f7b1015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 -- 172.21.15.90:0/2238547366 >> v1:172.21.15.102:6789/0 conn(0x7f7b101449a0 legacy=0x7f7b100b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.704+0000 7f7b0d400640 1 -- 172.21.15.90:0/2238547366 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1015b7d0 con 0x7f7b1015bfd0 2024-03-31T15:55:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b0d400640 1 --2- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b1006b820 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0097c0 tx=0x7f7afc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9786213fefcbc378 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b0ca00640 1 -- 172.21.15.90:0/2238547366 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0b0070 con 0x7f7b1015bfd0 2024-03-31T15:55:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b0ca00640 1 -- 172.21.15.90:0/2238547366 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7afc0ab600 con 0x7f7b1015bfd0 2024-03-31T15:55:58.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b0ca00640 1 -- 172.21.15.90:0/2238547366 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0ab9d0 con 0x7f7b1015bfd0 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 -- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 msgr2=0x7f7b1006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 --2- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b1006b820 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0097c0 tx=0x7f7afc01c4a0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 -- 172.21.15.90:0/2238547366 shutdown_connections 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 --2- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b1006bd60 0x7f7b1015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 --2- 172.21.15.90:0/2238547366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b1006b820 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 -- 172.21.15.90:0/2238547366 >> 172.21.15.90:0/2238547366 conn(0x7f7b100b1c80 msgr2=0x7f7b100b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 -- 172.21.15.90:0/2238547366 shutdown_connections 2024-03-31T15:55:58.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.708+0000 7f7b14c00640 1 -- 172.21.15.90:0/2238547366 wait complete. 2024-03-31T15:55:58.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 Processor -- start 2024-03-31T15:55:58.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- start start 2024-03-31T15:55:58.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 0x7f7b100d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b10159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b10167df0 con 0x7f7b1006bd60 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1015b450 con 0x7f7b1015bfd0 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b10167f70 con 0x7f7b101449a0 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45784/0 (socket says 172.21.15.90:45784) 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 -- 172.21.15.90:0/18467307 learned_addr learned my addr 172.21.15.90:0/18467307 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0e800640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b10159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0d400640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 0x7f7b100d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 -- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 msgr2=0x7f7b100d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 0x7f7b100d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 -- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 msgr2=0x7f7b10159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b10159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 -- 172.21.15.90:0/18467307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b100cee40 con 0x7f7b1006bd60 2024-03-31T15:55:58.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0d400640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 0x7f7b100d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:58.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b0de00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000c420 tx=0x7f7b0000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:58.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00017070 con 0x7f7b1006bd60 2024-03-31T15:55:58.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b00005cf0 con 0x7f7b1006bd60 2024-03-31T15:55:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b000072f0 con 0x7f7b1006bd60 2024-03-31T15:55:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1015a0f0 con 0x7f7b1006bd60 2024-03-31T15:55:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.712+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b101614e0 con 0x7f7b1006bd60 2024-03-31T15:55:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b100686d0 con 0x7f7b1006bd60 2024-03-31T15:55:58.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b0001a070 con 0x7f7b1006bd60 2024-03-31T15:55:58.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b06000640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 0x7f7af006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:58.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f7b00085f90 con 0x7f7b1006bd60 2024-03-31T15:55:58.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b0d400640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 0x7f7af006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:58.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.716+0000 7f7b0d400640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 0x7f7af006a840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0ad000 tx=0x7f7afc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:58.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.724+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b00055c40 con 0x7f7b1006bd60 2024-03-31T15:55:58.895 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.900+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b10072050 con 0x7f7b1006bd60 2024-03-31T15:55:58.896 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b06000640 1 -- 172.21.15.90:0/18467307 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f7b00059f40 con 0x7f7b1006bd60 2024-03-31T15:55:58.897 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:55:58.897 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:55:58.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 msgr2=0x7f7af006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 0x7f7af006a840 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0ad000 tx=0x7f7afc0a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 msgr2=0x7f7b1015a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000c420 tx=0x7f7b0000c8e0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 shutdown_connections 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7af0068380 0x7f7af006a840 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b1015bfd0 0x7f7b10159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b101449a0 0x7f7b100d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 --2- 172.21.15.90:0/18467307 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b1006bd60 0x7f7b1015a310 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 >> 172.21.15.90:0/18467307 conn(0x7f7b100b1c80 msgr2=0x7f7b100b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 shutdown_connections 2024-03-31T15:55:58.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:58.904+0000 7f7b14c00640 1 -- 172.21.15.90:0/18467307 wait complete. 2024-03-31T15:55:58.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:55:58.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:55:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 Processor -- start 2024-03-31T15:55:59.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 -- start start 2024-03-31T15:55:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf2406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbf240cdf20 0x7fbf2406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf2415e8c0 con 0x7fbf2406ce50 2024-03-31T15:55:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbf240d0ff0 con 0x7fbf240cdf20 2024-03-31T15:55:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf2b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbf240d12d0 con 0x7fbf2415e4e0 2024-03-31T15:55:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf23e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf2406ce50 0x7fbf240ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45164/0 (socket says 172.21.15.90:45164) 2024-03-31T15:55:59.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.064+0000 7fbf23e00640 1 -- 172.21.15.90:0/9426181 learned_addr learned my addr 172.21.15.90:0/9426181 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf29400640 1 --2- 172.21.15.90:0/9426181 >> v2:172.21.15.102:3300/0 conn(0x7fbf240cdf20 0x7fbf2406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf2406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953366294 0 0) 0x7fbf2415e8c0 con 0x7fbf2406ce50 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf0c003530 con 0x7fbf2406ce50 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3374783628 0 0) 0x7fbf0c003530 con 0x7fbf2406ce50 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 msgr2=0x7fbf2406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf2406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 >> v2:172.21.15.102:3300/0 conn(0x7fbf240cdf20 msgr2=0x7fbf2406d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 --2- 172.21.15.90:0/9426181 >> v2:172.21.15.102:3300/0 conn(0x7fbf240cdf20 0x7fbf2406d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf240cc710 con 0x7fbf2406ce50 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fbf10001680 con 0x7fbf2406ce50 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 >> v1:172.21.15.90:6789/0 conn(0x7fbf2406ce50 legacy=0x7fbf240ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0039c0 0x7fbf0c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c004d50 0x7fbf0c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf240d0ff0 con 0x7fbf0c0039c0 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbf240d12d0 con 0x7fbf0c004d50 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0c00ba00 con 0x7fbf2415e4e0 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0039c0 0x7fbf0c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf23e00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf29400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c004d50 0x7fbf0c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 -- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 msgr2=0x7fbf0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf2415e4e0 0x7fbf0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 -- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c004d50 msgr2=0x7fbf0c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c004d50 0x7fbf0c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf28a00640 1 -- 172.21.15.90:0/9426181 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c003530 con 0x7fbf0c0039c0 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.068+0000 7fbf29400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c004d50 0x7fbf0c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf28a00640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0039c0 0x7fbf0c003de0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009ae0 tx=0x7fbf1801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f2ec6d3f3516b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf180096b0 con 0x7fbf0c0039c0 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf180a6450 con 0x7fbf0c0039c0 2024-03-31T15:55:59.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf23400640 1 -- 172.21.15.90:0/9426181 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf180a6840 con 0x7fbf0c0039c0 2024-03-31T15:55:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- 172.21.15.90:0/9426181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0039c0 msgr2=0x7fbf0c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 --2- 172.21.15.90:0/9426181 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf0c0039c0 0x7fbf0c003de0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009ae0 tx=0x7fbf1801ca70 comp rx=0 tx=0).stop 2024-03-31T15:55:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf29400640 1 -- 172.21.15.90:0/9426181 reap_dead start 2024-03-31T15:55:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- 172.21.15.90:0/9426181 shutdown_connections 2024-03-31T15:55:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- 172.21.15.90:0/9426181 >> 172.21.15.90:0/9426181 conn(0x7fbf240d1f20 msgr2=0x7fbf240d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- 172.21.15.90:0/9426181 shutdown_connections 2024-03-31T15:55:59.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- 172.21.15.90:0/9426181 wait complete. 2024-03-31T15:55:59.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 Processor -- start 2024-03-31T15:55:59.067 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 -- start start 2024-03-31T15:55:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c0039c0 0x7fbf24165b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.072+0000 7fbf2b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf241f7820 0x7fbf24147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf241691a0 con 0x7fbf24156cd0 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbf24169020 con 0x7fbf0c0039c0 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbf24169320 con 0x7fbf241f7820 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf28a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c0039c0 0x7fbf24165b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45804/0 (socket says 172.21.15.90:45804) 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 -- 172.21.15.90:0/3765713707 learned_addr learned my addr 172.21.15.90:0/3765713707 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf23e00640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf241f7820 0x7fbf24147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 -- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf241f7820 msgr2=0x7fbf24147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf241f7820 0x7fbf24147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 -- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c0039c0 msgr2=0x7fbf24165b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c0039c0 0x7fbf24165b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 -- 172.21.15.90:0/3765713707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf241575b0 con 0x7fbf24156cd0 2024-03-31T15:55:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf29400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbf240d3020 tx=0x7fbf1c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c009c10 con 0x7fbf24156cd0 2024-03-31T15:55:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf1c01c070 con 0x7fbf24156cd0 2024-03-31T15:55:59.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf24157890 con 0x7fbf24156cd0 2024-03-31T15:55:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c003be0 con 0x7fbf24156cd0 2024-03-31T15:55:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf241661b0 con 0x7fbf24156cd0 2024-03-31T15:55:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf240685f0 con 0x7fbf24156cd0 2024-03-31T15:55:59.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.076+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbf1c004030 con 0x7fbf24156cd0 2024-03-31T15:55:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.080+0000 7fbf20c00640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 0x7fbf0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.080+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fbf1c087480 con 0x7fbf24156cd0 2024-03-31T15:55:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.080+0000 7fbf28a00640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 0x7fbf0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.080+0000 7fbf28a00640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 0x7fbf0406a840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbf180ad000 tx=0x7fbf180b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.084+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbf1c010040 con 0x7fbf24156cd0 2024-03-31T15:55:59.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.240+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf240cec70 con 0x7fbf24156cd0 2024-03-31T15:55:59.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.240+0000 7fbf20c00640 1 -- 172.21.15.90:0/3765713707 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7fbf1c057020 con 0x7fbf24156cd0 2024-03-31T15:55:59.235 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 msgr2=0x7fbf0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 0x7fbf0406a840 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbf180ad000 tx=0x7fbf180b2040 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 msgr2=0x7fbf240c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbf240d3020 tx=0x7fbf1c00f580 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 shutdown_connections 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbf04068380 0x7fbf0406a840 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf24156cd0 0x7fbf240c7cb0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbf241f7820 0x7fbf24147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 --2- 172.21.15.90:0/3765713707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf0c0039c0 0x7fbf24165b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 >> 172.21.15.90:0/3765713707 conn(0x7fbf240d1f20 msgr2=0x7fbf240c6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 shutdown_connections 2024-03-31T15:55:59.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.244+0000 7fbf2b400640 1 -- 172.21.15.90:0/3765713707 wait complete. 2024-03-31T15:55:59.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:55:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 Processor -- start 2024-03-31T15:55:59.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 -- start start 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 0x7fa3c006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 0x7fa3c006d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c015e8c0 con 0x7fa3c00ce7a0 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa3c00d0ff0 con 0x7fa3c006ce50 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.400+0000 7fa3c4c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c00d12d0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3be800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa3c00ce7a0 0x7fa3c00cec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42726/0 (socket says 172.21.15.90:42726) 2024-03-31T15:55:59.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3be800640 1 -- 172.21.15.90:0/3264832722 learned_addr learned my addr 172.21.15.90:0/3264832722 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:55:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 --2- 172.21.15.90:0/3264832722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 0x7fa3c006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3b7e00640 1 --2- 172.21.15.90:0/3264832722 >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 0x7fa3c006d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bd400640 1 -- 172.21.15.90:0/3264832722 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353630048 0 0) 0x7fa3c015e8c0 con 0x7fa3c00ce7a0 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bd400640 1 -- 172.21.15.90:0/3264832722 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a4003530 con 0x7fa3c00ce7a0 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 -- 172.21.15.90:0/3264832722 >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 msgr2=0x7fa3c006d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 --2- 172.21.15.90:0/3264832722 >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 0x7fa3c006d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 -- 172.21.15.90:0/3264832722 >> v1:172.21.15.102:6789/0 conn(0x7fa3c00ce7a0 legacy=0x7fa3c00cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 -- 172.21.15.90:0/3264832722 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c00cc710 con 0x7fa3c015e4e0 2024-03-31T15:55:59.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3b7e00640 1 --2- 172.21.15.90:0/3264832722 >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 0x7fa3c006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:55:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.404+0000 7fa3bde00640 1 --2- 172.21.15.90:0/3264832722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 0x7fa3c006c910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097c0 tx=0x7fa3b001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2abe9f69e6d8937 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3bd400640 1 -- 172.21.15.90:0/3264832722 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00b0070 con 0x7fa3c015e4e0 2024-03-31T15:55:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3bd400640 1 -- 172.21.15.90:0/3264832722 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b00ab600 con 0x7fa3c015e4e0 2024-03-31T15:55:59.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3bd400640 1 -- 172.21.15.90:0/3264832722 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00ab9b0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- 172.21.15.90:0/3264832722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 msgr2=0x7fa3c006c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/3264832722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 0x7fa3c006c910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097c0 tx=0x7fa3b001c410 comp rx=0 tx=0).stop 2024-03-31T15:55:59.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- 172.21.15.90:0/3264832722 shutdown_connections 2024-03-31T15:55:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/3264832722 >> v2:172.21.15.102:3300/0 conn(0x7fa3c006ce50 0x7fa3c006d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/3264832722 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c015e4e0 0x7fa3c006c910 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- 172.21.15.90:0/3264832722 >> 172.21.15.90:0/3264832722 conn(0x7fa3c00d1f20 msgr2=0x7fa3c00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- 172.21.15.90:0/3264832722 shutdown_connections 2024-03-31T15:55:59.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- 172.21.15.90:0/3264832722 wait complete. 2024-03-31T15:55:59.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 Processor -- start 2024-03-31T15:55:59.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.408+0000 7fa3c4c00640 1 -- start start 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3c006ce50 0x7fa3c00cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c00ce7a0 0x7fa3c01541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00d0d90 con 0x7fa3c015e4e0 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00d0c10 con 0x7fa3c006ce50 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c016b840 con 0x7fa3c00ce7a0 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45810/0 (socket says 172.21.15.90:45810) 2024-03-31T15:55:59.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 -- 172.21.15.90:0/662470149 learned_addr learned my addr 172.21.15.90:0/662470149 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3bde00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3c006ce50 0x7fa3c00cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3b7e00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c00ce7a0 0x7fa3c01541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 -- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c00ce7a0 msgr2=0x7fa3c01541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c00ce7a0 0x7fa3c01541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 -- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3c006ce50 msgr2=0x7fa3c00cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3c006ce50 0x7fa3c00cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 -- 172.21.15.90:0/662470149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c0157060 con 0x7fa3c015e4e0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3be800640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800cb40 tx=0x7fa3b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8017070 con 0x7fa3c015e4e0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b800aca0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8012620 con 0x7fa3c015e4e0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c0154910 con 0x7fa3c015e4e0 2024-03-31T15:55:59.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c01575e0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.412+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c00685f0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.416+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa3b801a070 con 0x7fa3c015e4e0 2024-03-31T15:55:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.416+0000 7fa3b6000640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 0x7fa39c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.416+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fa3b8086a80 con 0x7fa3c015e4e0 2024-03-31T15:55:59.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.416+0000 7fa3bde00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 0x7fa39c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.416+0000 7fa3bde00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 0x7fa39c06a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00ad000 tx=0x7fa3b00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.420+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa3b8088e20 con 0x7fa3c015e4e0 2024-03-31T15:55:59.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.576+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3c015ad70 con 0x7fa3c015e4e0 2024-03-31T15:55:59.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.576+0000 7fa3b6000640 1 -- 172.21.15.90:0/662470149 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7fa3b80566b0 con 0x7fa3c015e4e0 2024-03-31T15:55:59.572 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:55:59.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 msgr2=0x7fa39c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 0x7fa39c06a840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00ad000 tx=0x7fa3b00a6000 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 msgr2=0x7fa3c01523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa3b800cb40 tx=0x7fa3b800a060 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 shutdown_connections 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa39c068380 0x7fa39c06a840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3c015e4e0 0x7fa3c01523a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa3c00ce7a0 0x7fa3c01541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 --2- 172.21.15.90:0/662470149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3c006ce50 0x7fa3c00cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 >> 172.21.15.90:0/662470149 conn(0x7fa3c00d1f20 msgr2=0x7fa3c00c5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 shutdown_connections 2024-03-31T15:55:59.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.580+0000 7fa3c4c00640 1 -- 172.21.15.90:0/662470149 wait complete. 2024-03-31T15:55:59.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:55:59.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 Processor -- start 2024-03-31T15:55:59.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 -- start start 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff81c0cd640 0x7ff81c0af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81c0cefc0 con 0x7ff81c15c320 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff81c0aed70 con 0x7ff81c0cd640 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff820a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff81c0cf2a0 con 0x7ff81c0d2b20 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff81a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff81c0cd640 0x7ff81c0af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff81a800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff81c0cd640 0x7ff81c0af200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45820/0 (socket says 172.21.15.90:45820) 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff81a800640 1 -- 172.21.15.90:0/3140535684 learned_addr learned my addr 172.21.15.90:0/3140535684 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.736+0000 7ff811400640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff81a800640 1 -- 172.21.15.90:0/3140535684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 msgr2=0x7ff81c0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:55:59.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff81a800640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff81a800640 1 -- 172.21.15.90:0/3140535684 >> v1:172.21.15.102:6789/0 conn(0x7ff81c15c320 legacy=0x7ff81c0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff81a800640 1 -- 172.21.15.90:0/3140535684 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c0cf580 con 0x7ff81c0cd640 2024-03-31T15:55:59.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff811400640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:55:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff81a800640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c0cd640 0x7ff81c0af200 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff814009e50 tx=0x7ff81401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d30c064c28d66eb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff819400640 1 -- 172.21.15.90:0/3140535684 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140a5770 con 0x7ff81c0cd640 2024-03-31T15:55:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff819400640 1 -- 172.21.15.90:0/3140535684 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c0d0f20 con 0x7ff81c0cd640 2024-03-31T15:55:59.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff819400640 1 -- 172.21.15.90:0/3140535684 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140a5b30 con 0x7ff81c0cd640 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 -- 172.21.15.90:0/3140535684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c0cd640 msgr2=0x7ff81c0af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c0cd640 0x7ff81c0af200 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff814009e50 tx=0x7ff81401c9a0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 -- 172.21.15.90:0/3140535684 shutdown_connections 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c0cd640 0x7ff81c0af200 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 --2- 172.21.15.90:0/3140535684 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.740+0000 7ff820a00640 1 -- 172.21.15.90:0/3140535684 >> 172.21.15.90:0/3140535684 conn(0x7ff81c0d1f20 msgr2=0x7ff81c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- 172.21.15.90:0/3140535684 shutdown_connections 2024-03-31T15:55:59.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- 172.21.15.90:0/3140535684 wait complete. 2024-03-31T15:55:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 Processor -- start 2024-03-31T15:55:59.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- start start 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff81c0cd640 0x7ff81c0b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c0b0360 con 0x7ff81c15c320 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c0b01e0 con 0x7ff81c0cd640 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff81c0b04e0 con 0x7ff81c0d2b20 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45832/0 (socket says 172.21.15.90:45832) 2024-03-31T15:55:59.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 -- 172.21.15.90:0/2805805922 learned_addr learned my addr 172.21.15.90:0/2805805922 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff811400640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff819e00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff81c0cd640 0x7ff81c0b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 -- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 msgr2=0x7ff81c153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c153450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 -- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff81c0cd640 msgr2=0x7ff81c0b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff81c0cd640 0x7ff81c0b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 -- 172.21.15.90:0/2805805922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c0c66d0 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff81a800640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff8140af040 tx=0x7ff814019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140bc020 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8140b8690 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c0c8900 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.744+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814009510 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c0c6c50 con 0x7ff81c15c320 2024-03-31T15:55:59.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c0685f0 con 0x7ff81c15c320 2024-03-31T15:55:59.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff8140bf020 con 0x7ff81c15c320 2024-03-31T15:55:59.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff812a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 0x7ff7f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:55:59.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ff814121bd0 con 0x7ff81c15c320 2024-03-31T15:55:59.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.748+0000 7ff819e00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 0x7ff7f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:55:59.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.752+0000 7ff819e00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 0x7ff7f806a840 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff80c001810 tx=0x7ff80c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:55:59.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.752+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff8140c6030 con 0x7ff81c15c320 2024-03-31T15:55:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.912+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff81c15b7c0 con 0x7ff81c15c320 2024-03-31T15:55:59.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.912+0000 7ff812a00640 1 -- 172.21.15.90:0/2805805922 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7ff814016620 con 0x7ff81c15c320 2024-03-31T15:55:59.907 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 msgr2=0x7ff7f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 0x7ff7f806a840 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff80c001810 tx=0x7ff80c009040 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 msgr2=0x7ff81c153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff8140af040 tx=0x7ff814019740 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 shutdown_connections 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7f8068380 0x7ff7f806a840 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff81c15c320 0x7ff81c153ca0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff81c0d2b20 0x7ff81c153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 --2- 172.21.15.90:0/2805805922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff81c0cd640 0x7ff81c0b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 >> 172.21.15.90:0/2805805922 conn(0x7ff81c0d1f20 msgr2=0x7ff81c06db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 shutdown_connections 2024-03-31T15:55:59.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:55:59.916+0000 7ff820a00640 1 -- 172.21.15.90:0/2805805922 wait complete. 2024-03-31T15:55:59.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:56:00.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.072+0000 7f8df5200640 1 Processor -- start 2024-03-31T15:56:00.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.072+0000 7f8df5200640 1 -- start start 2024-03-31T15:56:00.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8df5200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df015c320 0x7f8df00cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8df5200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8df00cd640 0x7f8df00af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8df5200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00cefc0 con 0x7f8df00d2b20 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8df5200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8df00aed70 con 0x7f8df015c320 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8df5200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8df00cf2a0 con 0x7f8df00cd640 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df015c320 0x7f8df00cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8df015c320 0x7f8df00cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45834/0 (socket says 172.21.15.90:45834) 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 -- 172.21.15.90:0/309477383 learned_addr learned my addr 172.21.15.90:0/309477383 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dee800640 1 --2- 172.21.15.90:0/309477383 >> v2:172.21.15.102:3300/0 conn(0x7f8df00cd640 0x7f8df00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 -- 172.21.15.90:0/309477383 >> v2:172.21.15.102:3300/0 conn(0x7f8df00cd640 msgr2=0x7f8df00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 --2- 172.21.15.90:0/309477383 >> v2:172.21.15.102:3300/0 conn(0x7f8df00cd640 0x7f8df00af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 -- 172.21.15.90:0/309477383 >> v1:172.21.15.102:6789/0 conn(0x7f8df00d2b20 legacy=0x7f8df00ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.070 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.076+0000 7f8dede00640 1 -- 172.21.15.90:0/309477383 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df00cf580 con 0x7f8df015c320 2024-03-31T15:56:00.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8dede00640 1 --2- 172.21.15.90:0/309477383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df00cc9f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8df006c520 tx=0x7f8de001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7321030c2c1bc3ce server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8deca00640 1 -- 172.21.15.90:0/309477383 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00b1070 con 0x7f8df015c320 2024-03-31T15:56:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8deca00640 1 -- 172.21.15.90:0/309477383 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df00d0f20 con 0x7f8df015c320 2024-03-31T15:56:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8deca00640 1 -- 172.21.15.90:0/309477383 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00ac820 con 0x7f8df015c320 2024-03-31T15:56:00.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 -- 172.21.15.90:0/309477383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 msgr2=0x7f8df00cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 --2- 172.21.15.90:0/309477383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df00cc9f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8df006c520 tx=0x7f8de001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 -- 172.21.15.90:0/309477383 shutdown_connections 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 --2- 172.21.15.90:0/309477383 >> v2:172.21.15.102:3300/0 conn(0x7f8df00cd640 0x7f8df00af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 --2- 172.21.15.90:0/309477383 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df00cc9f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 -- 172.21.15.90:0/309477383 >> 172.21.15.90:0/309477383 conn(0x7f8df00d1f20 msgr2=0x7f8df00d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 -- 172.21.15.90:0/309477383 shutdown_connections 2024-03-31T15:56:00.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.080+0000 7f8df5200640 1 -- 172.21.15.90:0/309477383 wait complete. 2024-03-31T15:56:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 Processor -- start 2024-03-31T15:56:00.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- start start 2024-03-31T15:56:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8df00cd640 0x7f8df0157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 0x7f8df0165e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8df00b02e0 con 0x7f8df015c320 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8df00b0160 con 0x7f8df00d2b20 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8df00b0460 con 0x7f8df00cd640 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45844/0 (socket says 172.21.15.90:45844) 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 -- 172.21.15.90:0/1870289712 learned_addr learned my addr 172.21.15.90:0/1870289712 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dede00640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8df00cd640 0x7f8df0157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8ded400640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 0x7f8df0165e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 -- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8df00cd640 msgr2=0x7f8df0157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8df00cd640 0x7f8df0157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 -- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 msgr2=0x7f8df0165e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 0x7f8df0165e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.080 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 -- 172.21.15.90:0/1870289712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df0156220 con 0x7f8df015c320 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8ded400640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 0x7f8df0165e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8dee800640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8de800efc0 tx=0x7f8de800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8018020 con 0x7f8df015c320 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8014480 con 0x7f8df015c320 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8009e60 con 0x7f8df015c320 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df015a600 con 0x7f8df015c320 2024-03-31T15:56:00.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.084+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df01564c0 con 0x7f8df015c320 2024-03-31T15:56:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df00685f0 con 0x7f8df015c320 2024-03-31T15:56:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8de801b020 con 0x7f8df015c320 2024-03-31T15:56:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8de6000640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 0x7f8dd006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f8de8087a00 con 0x7f8df015c320 2024-03-31T15:56:00.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8dede00640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 0x7f8dd006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.088+0000 7f8dede00640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 0x7f8dd006a840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8de00ae000 tx=0x7f8de00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.092+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8de8022030 con 0x7f8df015c320 2024-03-31T15:56:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.248+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f8df0158710 con 0x7f8df015c320 2024-03-31T15:56:00.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8de6000640 1 -- 172.21.15.90:0/1870289712 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f8de8010040 con 0x7f8df015c320 2024-03-31T15:56:00.245 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 msgr2=0x7f8dd006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 0x7f8dd006a840 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8de00ae000 tx=0x7f8de00a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 msgr2=0x7f8df0163fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8de800efc0 tx=0x7f8de800c490 comp rx=0 tx=0).stop 2024-03-31T15:56:00.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 shutdown_connections 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dd0068380 0x7f8dd006a840 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8df015c320 0x7f8df0163fe0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8df00d2b20 0x7f8df0165e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 --2- 172.21.15.90:0/1870289712 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8df00cd640 0x7f8df0157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 >> 172.21.15.90:0/1870289712 conn(0x7f8df00d1f20 msgr2=0x7f8df0074a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 shutdown_connections 2024-03-31T15:56:00.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.252+0000 7f8df5200640 1 -- 172.21.15.90:0/1870289712 wait complete. 2024-03-31T15:56:00.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:00.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 Processor -- start 2024-03-31T15:56:00.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 -- start start 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff71015d240 0x7ff7100ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff710165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7100d0ea0 con 0x7ff7100ce910 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff7100cd460 con 0x7ff71015d240 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff715400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff7100cd740 con 0x7ff710163bd0 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70e800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff7100ce910 0x7ff7100cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45180/0 (socket says 172.21.15.90:45180) 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70e800640 1 -- 172.21.15.90:0/678666654 learned_addr learned my addr 172.21.15.90:0/678666654 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161396311 0 0) 0x7ff7100d0ea0 con 0x7ff7100ce910 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6f4003530 con 0x7ff7100ce910 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff707e00640 1 --2- 172.21.15.90:0/678666654 >> v2:172.21.15.102:3300/0 conn(0x7ff71015d240 0x7ff7100ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70f200640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff710165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1154232960 0 0) 0x7ff6f4003530 con 0x7ff7100ce910 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 msgr2=0x7ff710165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff710165fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 >> v2:172.21.15.102:3300/0 conn(0x7ff71015d240 msgr2=0x7ff7100ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 --2- 172.21.15.90:0/678666654 >> v2:172.21.15.102:3300/0 conn(0x7ff71015d240 0x7ff7100ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7100cda20 con 0x7ff7100ce910 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.408+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7ff700001680 con 0x7ff7100ce910 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 >> v1:172.21.15.90:6789/0 conn(0x7ff7100ce910 legacy=0x7ff7100cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6f40039c0 0x7ff6f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6f4004810 0x7ff6f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff6f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7100cd460 con 0x7ff6f40039c0 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7100cd740 con 0x7ff6f4004810 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f400bb30 con 0x7ff710163bd0 2024-03-31T15:56:00.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6f40039c0 0x7ff6f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff707e00640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6f4004810 0x7ff6f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70f200640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff6f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 -- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 msgr2=0x7ff6f40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff710163bd0 0x7ff6f40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 -- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6f4004810 msgr2=0x7ff6f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6f4004810 0x7ff6f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 -- 172.21.15.90:0/678666654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f4003530 con 0x7ff6f40039c0 2024-03-31T15:56:00.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70e800640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6f40039c0 0x7ff6f4003de0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff70000c290 tx=0x7ff70001ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a83d084ddd01b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700007d60 con 0x7ff6f40039c0 2024-03-31T15:56:00.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7000a9020 con 0x7ff6f40039c0 2024-03-31T15:56:00.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.412+0000 7ff70de00640 1 -- 172.21.15.90:0/678666654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700005ae0 con 0x7ff6f40039c0 2024-03-31T15:56:00.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- 172.21.15.90:0/678666654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6f40039c0 msgr2=0x7ff6f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 --2- 172.21.15.90:0/678666654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6f40039c0 0x7ff6f4003de0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff70000c290 tx=0x7ff70001ccd0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 -- 172.21.15.90:0/678666654 reap_dead start 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- 172.21.15.90:0/678666654 shutdown_connections 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- 172.21.15.90:0/678666654 >> 172.21.15.90:0/678666654 conn(0x7ff71008bfd0 msgr2=0x7ff7100d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- 172.21.15.90:0/678666654 shutdown_connections 2024-03-31T15:56:00.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- 172.21.15.90:0/678666654 wait complete. 2024-03-31T15:56:00.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 Processor -- start 2024-03-31T15:56:00.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- start start 2024-03-31T15:56:00.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 0x7ff71006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7101462d0 0x7ff7100ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff710166950 con 0x7ff7100b2e80 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7101667d0 con 0x7ff7101462d0 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff715400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff710166ad0 con 0x7ff6f40039c0 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45870/0 (socket says 172.21.15.90:45870) 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 -- 172.21.15.90:0/435170197 learned_addr learned my addr 172.21.15.90:0/435170197 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70e800640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 0x7ff71006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff707e00640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7101462d0 0x7ff7100ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 -- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 msgr2=0x7ff71006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 0x7ff71006e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 -- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7101462d0 msgr2=0x7ff7100ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7101462d0 0x7ff7100ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70f200640 1 -- 172.21.15.90:0/435170197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7100cb1a0 con 0x7ff7100b2e80 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.416+0000 7ff70e800640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 0x7ff71006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff70f200640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7100cc600 tx=0x7ff708010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708010d60 con 0x7ff7100b2e80 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff70801c070 con 0x7ff7100b2e80 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7100cb480 con 0x7ff7100b2e80 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708017670 con 0x7ff7100b2e80 2024-03-31T15:56:00.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7100cba40 con 0x7ff7100b2e80 2024-03-31T15:56:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff710068660 con 0x7ff7100b2e80 2024-03-31T15:56:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff708017810 con 0x7ff7100b2e80 2024-03-31T15:56:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 0x7ff6ec06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ff708087920 con 0x7ff7100b2e80 2024-03-31T15:56:00.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.420+0000 7ff70e800640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 0x7ff6ec06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.424+0000 7ff70e800640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 0x7ff6ec06a870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff7000aa000 tx=0x7ff700001a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.424+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff708026020 con 0x7ff7100b2e80 2024-03-31T15:56:00.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff710157ee0 con 0x7ff7100b2e80 2024-03-31T15:56:00.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff706a00640 1 -- 172.21.15.90:0/435170197 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7ff708057b20 con 0x7ff7100b2e80 2024-03-31T15:56:00.593 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:00.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 msgr2=0x7ff6ec06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 0x7ff6ec06a870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff7000aa000 tx=0x7ff700001a50 comp rx=0 tx=0).stop 2024-03-31T15:56:00.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 msgr2=0x7ff7100b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff7100cc600 tx=0x7ff708010530 comp rx=0 tx=0).stop 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 shutdown_connections 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff6ec0683b0 0x7ff6ec06a870 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7100b2e80 0x7ff7100b3300 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7101462d0 0x7ff7100ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 --2- 172.21.15.90:0/435170197 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff6f40039c0 0x7ff71006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 >> 172.21.15.90:0/435170197 conn(0x7ff71008bfd0 msgr2=0x7ff710070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 shutdown_connections 2024-03-31T15:56:00.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.600+0000 7ff715400640 1 -- 172.21.15.90:0/435170197 wait complete. 2024-03-31T15:56:00.614 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:00.614 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:00.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:00.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.808+0000 7f7fc4400640 1 Processor -- start 2024-03-31T15:56:00.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 -- start start 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fbc15e1b0 0x7f7fbc162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 0x7f7fbc06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7fbc1622b0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7fbc06da60 con 0x7f7fbc06c7b0 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc4400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc15f780 con 0x7f7fbc06d5f0 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fbc15e1b0 0x7f7fbc162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fbc15e1b0 0x7f7fbc162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45886/0 (socket says 172.21.15.90:45886) 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 -- 172.21.15.90:0/666202221 learned_addr learned my addr 172.21.15.90:0/666202221 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 -- 172.21.15.90:0/666202221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 msgr2=0x7f7fbc15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc2400640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1000640 1 --2- 172.21.15.90:0/666202221 >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 0x7f7fbc06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 -- 172.21.15.90:0/666202221 >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 msgr2=0x7f7fbc06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/666202221 >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 0x7f7fbc06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1a00640 1 -- 172.21.15.90:0/666202221 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc15f1c0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc2400640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:00.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.812+0000 7f7fc1000640 1 --2- 172.21.15.90:0/666202221 >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 0x7f7fbc06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:00.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc162740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc06ed50 tx=0x7f7fb001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b39144d13e3e47b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fbbe00640 1 -- 172.21.15.90:0/666202221 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00ab620 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fbbe00640 1 -- 172.21.15.90:0/666202221 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb00b0070 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fbbe00640 1 -- 172.21.15.90:0/666202221 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00ab9e0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 -- 172.21.15.90:0/666202221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 msgr2=0x7f7fbc162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc162740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc06ed50 tx=0x7f7fb001c460 comp rx=0 tx=0).stop 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 -- 172.21.15.90:0/666202221 shutdown_connections 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 --2- 172.21.15.90:0/666202221 >> v2:172.21.15.102:3300/0 conn(0x7f7fbc06c7b0 0x7f7fbc06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 --2- 172.21.15.90:0/666202221 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc162740 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 -- 172.21.15.90:0/666202221 >> 172.21.15.90:0/666202221 conn(0x7f7fbc09aa40 msgr2=0x7f7fbc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 -- 172.21.15.90:0/666202221 shutdown_connections 2024-03-31T15:56:00.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.816+0000 7f7fc4400640 1 -- 172.21.15.90:0/666202221 wait complete. 2024-03-31T15:56:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 Processor -- start 2024-03-31T15:56:00.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 -- start start 2024-03-31T15:56:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fbc06c7b0 0x7f7fbc0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc0d27c0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc162c80 con 0x7f7fbc06c7b0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc4400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc0d2940 con 0x7f7fbc06d5f0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc2400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc2400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45890/0 (socket says 172.21.15.90:45890) 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc2400640 1 -- 172.21.15.90:0/718421527 learned_addr learned my addr 172.21.15.90:0/718421527 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.820+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fbc06c7b0 0x7f7fbc0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc1000640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 -- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 msgr2=0x7f7fbc151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 -- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fbc06c7b0 msgr2=0x7f7fbc0b0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fbc06c7b0 0x7f7fbc0b0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 -- 172.21.15.90:0/718421527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc154210 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc1000640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc2400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400cef0 tx=0x7f7fb400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb401a070 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb400d860 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400dc50 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc1544f0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc154ab0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fbc0686d0 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7fb401d070 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.824+0000 7f7fb9600640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 0x7f7f9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:00.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.828+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f7fb4087560 con 0x7f7fbc15e1b0 2024-03-31T15:56:00.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.828+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 0x7f7f9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:00.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.828+0000 7f7fc1a00640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 0x7f7f9c06a840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00af000 tx=0x7f7fb00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:00.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:00.832+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7fb40577e0 con 0x7f7fbc15e1b0 2024-03-31T15:56:01.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.008+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fbc159de0 con 0x7f7fbc15e1b0 2024-03-31T15:56:01.004 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fb9600640 1 -- 172.21.15.90:0/718421527 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f7fb40570e0 con 0x7f7fbc15e1b0 2024-03-31T15:56:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:01.004 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 msgr2=0x7f7f9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 0x7f7f9c06a840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00af000 tx=0x7f7fb00a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 msgr2=0x7f7fbc155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7fb400cef0 tx=0x7f7fb400a710 comp rx=0 tx=0).stop 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 shutdown_connections 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f9c068380 0x7f7f9c06a840 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fbc15e1b0 0x7f7fbc155f20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fbc06d5f0 0x7f7fbc151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 --2- 172.21.15.90:0/718421527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fbc06c7b0 0x7f7fbc0b0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 >> 172.21.15.90:0/718421527 conn(0x7f7fbc09aa40 msgr2=0x7f7fbc0cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 shutdown_connections 2024-03-31T15:56:01.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.012+0000 7f7fc4400640 1 -- 172.21.15.90:0/718421527 wait complete. 2024-03-31T15:56:01.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:01.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:01.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 Processor -- start 2024-03-31T15:56:01.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 -- start start 2024-03-31T15:56:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f060815e4e0 0x7f060806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 0x7f060806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f060815e8c0 con 0x7f060806ce50 2024-03-31T15:56:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f06080d0ff0 con 0x7f060815e4e0 2024-03-31T15:56:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f060de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f06080d12d0 con 0x7f06080cdf20 2024-03-31T15:56:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f060815e4e0 0x7f060806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0607400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 0x7f060806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f060815e4e0 0x7f060806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:38420/0 (socket says 172.21.15.90:38420) 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 -- 172.21.15.90:0/565202424 learned_addr learned my addr 172.21.15.90:0/565202424 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f05ffe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f060806ce50 0x7f06080ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42746/0 (socket says 172.21.15.90:42746) 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606000640 1 -- 172.21.15.90:0/565202424 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1481509554 0 0) 0x7f060815e8c0 con 0x7f060806ce50 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606000640 1 -- 172.21.15.90:0/565202424 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05ec003530 con 0x7f060806ce50 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 -- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 msgr2=0x7f060806d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 0x7f060806d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 -- 172.21.15.90:0/565202424 >> v1:172.21.15.102:6789/0 conn(0x7f060806ce50 legacy=0x7f06080ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0606a00640 1 -- 172.21.15.90:0/565202424 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06080cc710 con 0x7f060815e4e0 2024-03-31T15:56:01.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.172+0000 7f0607400640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 0x7f060806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:01.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f0606a00640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060815e4e0 0x7f060806c910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f05f80098a0 tx=0x7f05f801c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574986925b1f3dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f0606000640 1 -- 172.21.15.90:0/565202424 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f80b1070 con 0x7f060815e4e0 2024-03-31T15:56:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f0606000640 1 -- 172.21.15.90:0/565202424 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f80ac620 con 0x7f060815e4e0 2024-03-31T15:56:01.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f0606000640 1 -- 172.21.15.90:0/565202424 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f80ac9f0 con 0x7f060815e4e0 2024-03-31T15:56:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 -- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060815e4e0 msgr2=0x7f060806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060815e4e0 0x7f060806c910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f05f80098a0 tx=0x7f05f801c4b0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 -- 172.21.15.90:0/565202424 shutdown_connections 2024-03-31T15:56:01.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f06080cdf20 0x7f060806d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 --2- 172.21.15.90:0/565202424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060815e4e0 0x7f060806c910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 -- 172.21.15.90:0/565202424 >> 172.21.15.90:0/565202424 conn(0x7f06080d1f20 msgr2=0x7f06080d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 -- 172.21.15.90:0/565202424 shutdown_connections 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.176+0000 7f060de00640 1 -- 172.21.15.90:0/565202424 wait complete. 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 Processor -- start 2024-03-31T15:56:01.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 -- start start 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f060806ce50 0x7f060809bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06080cdf20 0x7f060815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f060816b1b0 con 0x7f060815e4e0 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f060816b030 con 0x7f06080cdf20 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f060816b330 con 0x7f060806ce50 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45898/0 (socket says 172.21.15.90:45898) 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 -- 172.21.15.90:0/2010750059 learned_addr learned my addr 172.21.15.90:0/2010750059 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0606a00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f060806ce50 0x7f060809bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f05ffe00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06080cdf20 0x7f060815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 -- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f060806ce50 msgr2=0x7f060809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f060806ce50 0x7f060809bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 -- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06080cdf20 msgr2=0x7f060815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06080cdf20 0x7f060815c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 -- 172.21.15.90:0/2010750059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06080cbcb0 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f0607400640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f060000ef90 tx=0x7f0600009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600018020 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0600014480 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600014870 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.180+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06080cbf90 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f060815c440 con 0x7f060815e4e0 2024-03-31T15:56:01.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f060815cea0 con 0x7f060815e4e0 2024-03-31T15:56:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f060001b020 con 0x7f060815e4e0 2024-03-31T15:56:01.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f05ff400640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 0x7f05e406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f0600087a20 con 0x7f060815e4e0 2024-03-31T15:56:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f0606a00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 0x7f05e406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.184+0000 7f0606a00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 0x7f05e406a870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f05f80ae000 tx=0x7f05f80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.192+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0600022030 con 0x7f060815e4e0 2024-03-31T15:56:01.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f06080685f0 con 0x7f060815e4e0 2024-03-31T15:56:01.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f05ff400640 1 -- 172.21.15.90:0/2010750059 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7f0600010040 con 0x7f060815e4e0 2024-03-31T15:56:01.346 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 msgr2=0x7f05e406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 0x7f05e406a870 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f05f80ae000 tx=0x7f05f80a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 msgr2=0x7f060815b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f060000ef90 tx=0x7f0600009f90 comp rx=0 tx=0).stop 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 shutdown_connections 2024-03-31T15:56:01.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f05e40683b0 0x7f05e406a870 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f060815e4e0 0x7f060815b500 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06080cdf20 0x7f060815c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 --2- 172.21.15.90:0/2010750059 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f060806ce50 0x7f060809bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 >> 172.21.15.90:0/2010750059 conn(0x7f06080d1f20 msgr2=0x7f06080c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 shutdown_connections 2024-03-31T15:56:01.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.352+0000 7f060de00640 1 -- 172.21.15.90:0/2010750059 wait complete. 2024-03-31T15:56:01.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 Processor -- start 2024-03-31T15:56:01.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 -- start start 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb7c15e4e0 0x7fdb7c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c06d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 0x7fdb7c0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7c15e8c0 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7c0d0ff0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.512+0000 7fdb84800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c0d12d0 con 0x7fdb7c06ce50 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb7c15e4e0 0x7fdb7c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdb7c15e4e0 0x7fdb7c06c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45914/0 (socket says 172.21.15.90:45914) 2024-03-31T15:56:01.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 -- 172.21.15.90:0/557576498 learned_addr learned my addr 172.21.15.90:0/557576498 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 -- 172.21.15.90:0/557576498 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 msgr2=0x7fdb7c06d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb82800640 1 --2- 172.21.15.90:0/557576498 >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 0x7fdb7c0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81400640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c06d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c06d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 -- 172.21.15.90:0/557576498 >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 msgr2=0x7fdb7c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 --2- 172.21.15.90:0/557576498 >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 0x7fdb7c0cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 -- 172.21.15.90:0/557576498 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c0cc710 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb82800640 1 --2- 172.21.15.90:0/557576498 >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 0x7fdb7c0cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:01.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81400640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:01.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb81e00640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c06c910 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c075360 tx=0x7fdb7001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb11c186292d980d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb80a00640 1 -- 172.21.15.90:0/557576498 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700b0070 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb80a00640 1 -- 172.21.15.90:0/557576498 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb700ab600 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.516+0000 7fdb80a00640 1 -- 172.21.15.90:0/557576498 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700ab9d0 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- 172.21.15.90:0/557576498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c15e4e0 msgr2=0x7fdb7c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c06c910 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c075360 tx=0x7fdb7001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- 172.21.15.90:0/557576498 shutdown_connections 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- 172.21.15.90:0/557576498 >> v2:172.21.15.102:3300/0 conn(0x7fdb7c0ce7a0 0x7fdb7c0cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c06d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- 172.21.15.90:0/557576498 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c06c910 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- 172.21.15.90:0/557576498 >> 172.21.15.90:0/557576498 conn(0x7fdb7c0d1f20 msgr2=0x7fdb7c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- 172.21.15.90:0/557576498 shutdown_connections 2024-03-31T15:56:01.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- 172.21.15.90:0/557576498 wait complete. 2024-03-31T15:56:01.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 Processor -- start 2024-03-31T15:56:01.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- start start 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c1faa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c0cd090 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c0ccf10 con 0x7fdb7c15e4e0 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb84800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdb7c0cd210 con 0x7fdb7c06ce50 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45918/0 (socket says 172.21.15.90:45918) 2024-03-31T15:56:01.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 -- 172.21.15.90:0/1950897973 learned_addr learned my addr 172.21.15.90:0/1950897973 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81e00640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c1faa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 -- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 msgr2=0x7fdb7c1faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb82800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c1faa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 -- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 msgr2=0x7fdb7c15c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c15c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 -- 172.21.15.90:0/1950897973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c155630 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81e00640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c1faa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb82800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.520+0000 7fdb81400640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00c7c0 tx=0x7fdb6c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c007a30 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb6c007bd0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c0113f0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c155910 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c155ed0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c0685f0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdb6c011590 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 0x7fdb6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.524+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fdb6c086e90 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.528+0000 7fdb81e00640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 0x7fdb6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.528+0000 7fdb81e00640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 0x7fdb6006a840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ad000 tx=0x7fdb70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.528+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdb6c015020 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.684+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdb7c168e00 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.684+0000 7fdb76000640 1 -- 172.21.15.90:0/1950897973 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7fdb6c0571b0 con 0x7fdb7c0ce7a0 2024-03-31T15:56:01.679 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 msgr2=0x7fdb6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.681 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 0x7fdb6006a840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ad000 tx=0x7fdb70008040 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 msgr2=0x7fdb7c1fafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00c7c0 tx=0x7fdb6c00cc80 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 shutdown_connections 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fdb60068380 0x7fdb6006a840 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb7c15e4e0 0x7fdb7c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdb7c0ce7a0 0x7fdb7c1fafd0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 --2- 172.21.15.90:0/1950897973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdb7c06ce50 0x7fdb7c1faa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 >> 172.21.15.90:0/1950897973 conn(0x7fdb7c0d1f20 msgr2=0x7fdb7c0c5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 shutdown_connections 2024-03-31T15:56:01.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.688+0000 7fdb84800640 1 -- 172.21.15.90:0/1950897973 wait complete. 2024-03-31T15:56:01.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:01.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 Processor -- start 2024-03-31T15:56:01.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 -- start start 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 0x7f07f015b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f0072a60 con 0x7f07f0071940 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f00aeac0 con 0x7f07f00725a0 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07f6000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f07f00ae890 con 0x7f07f015b040 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07e7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f07f0071940 0x7f07f0071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:45182/0 (socket says 172.21.15.90:45182) 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07e7e00640 1 -- 172.21.15.90:0/2969929089 learned_addr learned my addr 172.21.15.90:0/2969929089 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07ef400640 1 --2- 172.21.15.90:0/2969929089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 0x7f07f015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3321103903 0 0) 0x7f07f0072a60 con 0x7f07f0071940 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07d4003530 con 0x7f07f0071940 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 438963911 0 0) 0x7f07f00aeac0 con 0x7f07f00725a0 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07f0072a60 con 0x7f07f00725a0 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07ef400640 1 -- 172.21.15.90:0/2969929089 >> v1:172.21.15.102:6789/0 conn(0x7f07f00725a0 legacy=0x7f07f00b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07ef400640 1 -- 172.21.15.90:0/2969929089 >> v1:172.21.15.90:6789/0 conn(0x7f07f0071940 legacy=0x7f07f0071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.848+0000 7f07ef400640 1 -- 172.21.15.90:0/2969929089 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f00aeda0 con 0x7f07f015b040 2024-03-31T15:56:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07ef400640 1 --2- 172.21.15.90:0/2969929089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 0x7f07f015b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07e00097f0 tx=0x7f07e001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c28ee82cde213a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00ab5f0 con 0x7f07f015b040 2024-03-31T15:56:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e00b0070 con 0x7f07f015b040 2024-03-31T15:56:01.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07eea00640 1 -- 172.21.15.90:0/2969929089 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00ab9b0 con 0x7f07f015b040 2024-03-31T15:56:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 -- 172.21.15.90:0/2969929089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 msgr2=0x7f07f015b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 --2- 172.21.15.90:0/2969929089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 0x7f07f015b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07e00097f0 tx=0x7f07e001c4e0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 -- 172.21.15.90:0/2969929089 shutdown_connections 2024-03-31T15:56:01.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 --2- 172.21.15.90:0/2969929089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f015b040 0x7f07f015b420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 -- 172.21.15.90:0/2969929089 >> 172.21.15.90:0/2969929089 conn(0x7f07f00b18b0 msgr2=0x7f07f00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 -- 172.21.15.90:0/2969929089 shutdown_connections 2024-03-31T15:56:01.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.852+0000 7f07f6000640 1 -- 172.21.15.90:0/2969929089 wait complete. 2024-03-31T15:56:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 Processor -- start 2024-03-31T15:56:01.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- start start 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f0071940 0x7f07f0144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 0x7f07f014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f07f006dc80 con 0x7f07f00725a0 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f07f006db00 con 0x7f07f015b040 2024-03-31T15:56:01.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f07f006de00 con 0x7f07f0071940 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45928/0 (socket says 172.21.15.90:45928) 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 -- 172.21.15.90:0/215953880 learned_addr learned my addr 172.21.15.90:0/215953880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07efe00640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 0x7f07f014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07ef400640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f0071940 0x7f07f0144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 -- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f0071940 msgr2=0x7f07f0144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f0071940 0x7f07f0144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 -- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 msgr2=0x7f07f014e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 0x7f07f014e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 -- 172.21.15.90:0/215953880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f014ff50 con 0x7f07f00725a0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07efe00640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 0x7f07f014e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7e00640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c500 tx=0x7f07d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8017070 con 0x7f07f00725a0 2024-03-31T15:56:01.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d8005cf0 con 0x7f07f00725a0 2024-03-31T15:56:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8007330 con 0x7f07f00725a0 2024-03-31T15:56:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f0150230 con 0x7f07f00725a0 2024-03-31T15:56:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f01507f0 con 0x7f07f00725a0 2024-03-31T15:56:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.856+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f0075880 con 0x7f07f00725a0 2024-03-31T15:56:01.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.860+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07d801a070 con 0x7f07f00725a0 2024-03-31T15:56:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.860+0000 7f07e7400640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 0x7f07cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.860+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f07d80861e0 con 0x7f07f00725a0 2024-03-31T15:56:01.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.860+0000 7f07ef400640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 0x7f07cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:01.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.860+0000 7f07ef400640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 0x7f07cc06a840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f07e00af000 tx=0x7f07e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:01.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:01.864+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f07d8056460 con 0x7f07f00725a0 2024-03-31T15:56:02.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.020+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f07f0151c60 con 0x7f07f00725a0 2024-03-31T15:56:02.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.020+0000 7f07e7400640 1 -- 172.21.15.90:0/215953880 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f07d8055d60 con 0x7f07f00725a0 2024-03-31T15:56:02.014 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 msgr2=0x7f07cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 0x7f07cc06a840 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f07e00af000 tx=0x7f07e00a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 msgr2=0x7f07f014e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f07d800c500 tx=0x7f07d800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 shutdown_connections 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f07cc068380 0x7f07cc06a840 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07f015b040 0x7f07f014e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07f00725a0 0x7f07f014e080 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 --2- 172.21.15.90:0/215953880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f07f0071940 0x7f07f0144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 >> 172.21.15.90:0/215953880 conn(0x7f07f00b18b0 msgr2=0x7f07f0147980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 shutdown_connections 2024-03-31T15:56:02.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.024+0000 7f07f6000640 1 -- 172.21.15.90:0/215953880 wait complete. 2024-03-31T15:56:02.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:56:02.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 Processor -- start 2024-03-31T15:56:02.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 -- start start 2024-03-31T15:56:02.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f838015c320 0x7f83800cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 0x7f83800af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83800cefc0 con 0x7f83800d2b20 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f83800aed70 con 0x7f838015c320 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8386a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f83800cf2a0 con 0x7f83800cd640 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f838015c320 0x7f83800cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f838015c320 0x7f83800cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45944/0 (socket says 172.21.15.90:45944) 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 -- 172.21.15.90:0/3712354949 learned_addr learned my addr 172.21.15.90:0/3712354949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 -- 172.21.15.90:0/3712354949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 msgr2=0x7f83800af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8384a00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 0x7f83800af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 0x7f83800af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 -- 172.21.15.90:0/3712354949 >> v1:172.21.15.102:6789/0 conn(0x7f83800d2b20 legacy=0x7f83800ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f837fe00640 1 -- 172.21.15.90:0/3712354949 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83800cf580 con 0x7f838015c320 2024-03-31T15:56:02.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.180+0000 7f8384a00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 0x7f83800af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:02.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f837fe00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f83800cc9f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8370009870 tx=0x7f837001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a4088e29cccf72 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f837ea00640 1 -- 172.21.15.90:0/3712354949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700b2070 con 0x7f838015c320 2024-03-31T15:56:02.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f837ea00640 1 -- 172.21.15.90:0/3712354949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83700a9070 con 0x7f838015c320 2024-03-31T15:56:02.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f837ea00640 1 -- 172.21.15.90:0/3712354949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700ad880 con 0x7f838015c320 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 -- 172.21.15.90:0/3712354949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 msgr2=0x7f83800cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f83800cc9f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8370009870 tx=0x7f837001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 -- 172.21.15.90:0/3712354949 shutdown_connections 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800cd640 0x7f83800af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 --2- 172.21.15.90:0/3712354949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f83800cc9f0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 -- 172.21.15.90:0/3712354949 >> 172.21.15.90:0/3712354949 conn(0x7f83800d1f20 msgr2=0x7f83800d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 -- 172.21.15.90:0/3712354949 shutdown_connections 2024-03-31T15:56:02.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.184+0000 7f8386a00640 1 -- 172.21.15.90:0/3712354949 wait complete. 2024-03-31T15:56:02.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 Processor -- start 2024-03-31T15:56:02.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- start start 2024-03-31T15:56:02.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83800cd640 0x7f83801581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800d2b20 0x7f8380159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f83800b0360 con 0x7f838015c320 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f83800b01e0 con 0x7f83800cd640 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f83800b04e0 con 0x7f83800d2b20 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45950/0 (socket says 172.21.15.90:45950) 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 -- 172.21.15.90:0/112171435 learned_addr learned my addr 172.21.15.90:0/112171435 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f837fe00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83800cd640 0x7f83801581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f837f400640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800d2b20 0x7f8380159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 -- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800d2b20 msgr2=0x7f8380159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800d2b20 0x7f8380159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 -- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83800cd640 msgr2=0x7f83801581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83800cd640 0x7f83801581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 -- 172.21.15.90:0/112171435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83800b1430 con 0x7f838015c320 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8384a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f837800cdb0 tx=0x7f837800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378003ba0 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8378003d60 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83800b1710 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378004260 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.188+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83800b1cd0 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83800685f0 con 0x7f838015c320 2024-03-31T15:56:02.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8378004400 con 0x7f838015c320 2024-03-31T15:56:02.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f8377e00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 0x7f836006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f8378087770 con 0x7f838015c320 2024-03-31T15:56:02.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f837fe00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 0x7f836006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.192+0000 7f837fe00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 0x7f836006a840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f83700af000 tx=0x7f83700a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.196+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8378014030 con 0x7f838015c320 2024-03-31T15:56:02.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.352+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f83800c5750 con 0x7f838015c320 2024-03-31T15:56:02.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.352+0000 7f8377e00640 1 -- 172.21.15.90:0/112171435 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f8378057310 con 0x7f838015c320 2024-03-31T15:56:02.346 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:02.348 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 msgr2=0x7f836006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 0x7f836006a840 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f83700af000 tx=0x7f83700a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 msgr2=0x7f8380159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f837800cdb0 tx=0x7f837800a650 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 shutdown_connections 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8360068380 0x7f836006a840 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f838015c320 0x7f8380159db0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f83800d2b20 0x7f8380159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 --2- 172.21.15.90:0/112171435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83800cd640 0x7f83801581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 >> 172.21.15.90:0/112171435 conn(0x7f83800d1f20 msgr2=0x7f8380070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 shutdown_connections 2024-03-31T15:56:02.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.356+0000 7f8386a00640 1 -- 172.21.15.90:0/112171435 wait complete. 2024-03-31T15:56:02.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.508+0000 7ff127000640 1 Processor -- start 2024-03-31T15:56:02.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 -- start start 2024-03-31T15:56:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 0x7ff120165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1200d0ea0 con 0x7ff12015d240 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff1200cd220 con 0x7ff120163bd0 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff127000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff1200cf230 con 0x7ff1200ce910 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff125000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 0x7ff120165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff125000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 0x7ff120165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:38482/0 (socket says 172.21.15.90:38482) 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff125000640 1 -- 172.21.15.90:0/1522344446 learned_addr learned my addr 172.21.15.90:0/1522344446 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:02.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:43104/0 (socket says 172.21.15.90:43104) 2024-03-31T15:56:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 -- 172.21.15.90:0/1522344446 >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 msgr2=0x7ff120165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 --2- 172.21.15.90:0/1522344446 >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 0x7ff120165fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 -- 172.21.15.90:0/1522344446 >> v1:172.21.15.102:6789/0 conn(0x7ff12015d240 legacy=0x7ff1200ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.512+0000 7ff11fe00640 1 -- 172.21.15.90:0/1522344446 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1200cd460 con 0x7ff1200ce910 2024-03-31T15:56:02.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff11fe00640 1 --2- 172.21.15.90:0/1522344446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1100097c0 tx=0x7ff11001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ec122c418dd7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff11f400640 1 -- 172.21.15.90:0/1522344446 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100ab600 con 0x7ff1200ce910 2024-03-31T15:56:02.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff11f400640 1 -- 172.21.15.90:0/1522344446 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1100b0070 con 0x7ff1200ce910 2024-03-31T15:56:02.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff11f400640 1 -- 172.21.15.90:0/1522344446 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100ab9c0 con 0x7ff1200ce910 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- 172.21.15.90:0/1522344446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 msgr2=0x7ff1200cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 --2- 172.21.15.90:0/1522344446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1100097c0 tx=0x7ff11001c3e0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- 172.21.15.90:0/1522344446 shutdown_connections 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 --2- 172.21.15.90:0/1522344446 >> v2:172.21.15.102:3300/0 conn(0x7ff120163bd0 0x7ff120165fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 --2- 172.21.15.90:0/1522344446 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff1200ce910 0x7ff1200cecf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- 172.21.15.90:0/1522344446 >> 172.21.15.90:0/1522344446 conn(0x7ff12008bfd0 msgr2=0x7ff1200d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:02.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- 172.21.15.90:0/1522344446 shutdown_connections 2024-03-31T15:56:02.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- 172.21.15.90:0/1522344446 wait complete. 2024-03-31T15:56:02.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 Processor -- start 2024-03-31T15:56:02.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.516+0000 7ff127000640 1 -- start start 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 0x7ff12014fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff120163bd0 0x7ff12015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff120168d20 con 0x7ff12015d240 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff12015d780 con 0x7ff1200ce910 2024-03-31T15:56:02.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff120168ea0 con 0x7ff120163bd0 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff11fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 0x7ff12014fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:45952/0 (socket says 172.21.15.90:45952) 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 -- 172.21.15.90:0/119391808 learned_addr learned my addr 172.21.15.90:0/119391808 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff125000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff120163bd0 0x7ff12015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 -- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff120163bd0 msgr2=0x7ff12015b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff120163bd0 0x7ff12015b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 -- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 msgr2=0x7ff12014fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 0x7ff12014fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 -- 172.21.15.90:0/119391808 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1201576b0 con 0x7ff12015d240 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff11fe00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 0x7ff12014fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff117e00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff10800c500 tx=0x7ff10800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108016020 con 0x7ff12015d240 2024-03-31T15:56:02.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff108005220 con 0x7ff12015d240 2024-03-31T15:56:02.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1200c65b0 con 0x7ff12015d240 2024-03-31T15:56:02.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108007030 con 0x7ff12015d240 2024-03-31T15:56:02.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff120150010 con 0x7ff12015d240 2024-03-31T15:56:02.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.520+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff120068660 con 0x7ff12015d240 2024-03-31T15:56:02.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.524+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1080071f0 con 0x7ff12015d240 2024-03-31T15:56:02.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.524+0000 7ff11cc00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 0x7ff0fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:02.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.524+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ff108086600 con 0x7ff12015d240 2024-03-31T15:56:02.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.524+0000 7ff11fe00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 0x7ff0fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:02.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.524+0000 7ff11fe00640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 0x7ff0fc06a840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff1100af000 tx=0x7ff1100a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:02.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.528+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff108020020 con 0x7ff12015d240 2024-03-31T15:56:02.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff1200d2670 con 0x7ff12015d240 2024-03-31T15:56:02.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff11cc00640 1 -- 172.21.15.90:0/119391808 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7ff1080561b0 con 0x7ff12015d240 2024-03-31T15:56:02.689 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 msgr2=0x7ff0fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 0x7ff0fc06a840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff1100af000 tx=0x7ff1100a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 msgr2=0x7ff1200c5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff10800c500 tx=0x7ff10800c9c0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 shutdown_connections 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0fc068380 0x7ff0fc06a840 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff120163bd0 0x7ff12015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff12015d240 0x7ff1200c5c40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 --2- 172.21.15.90:0/119391808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1200ce910 0x7ff12014fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 >> 172.21.15.90:0/119391808 conn(0x7ff12008bfd0 msgr2=0x7ff1200d1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 shutdown_connections 2024-03-31T15:56:02.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:02.696+0000 7ff127000640 1 -- 172.21.15.90:0/119391808 wait complete. 2024-03-31T15:56:02.711 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:56:02.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:02.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:02.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[39856]: fuse finished with error 0 and tester_r 0 2024-03-31T15:56:02.732 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:08.834 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:08.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:08.834 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:56:08.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-31T15:56:09.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 Processor -- start 2024-03-31T15:56:09.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 -- start start 2024-03-31T15:56:09.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f015e370 0x7fc2f006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2f006ea40 0x7fc2f006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00ceaa0 con 0x7fc2f00d00c0 2024-03-31T15:56:09.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f00ce290 con 0x7fc2f015e370 2024-03-31T15:56:09.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f00ce4e0 con 0x7fc2f006ea40 2024-03-31T15:56:09.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f015e370 0x7fc2f006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2efe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2f015e370 0x7fc2f006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52984/0 (socket says 172.21.15.90:52984) 2024-03-31T15:56:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2efe00640 1 -- 172.21.15.90:0/2172596947 learned_addr learned my addr 172.21.15.90:0/2172596947 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.036+0000 7fc2f5200640 1 --2- 172.21.15.90:0/2172596947 >> v2:172.21.15.102:3300/0 conn(0x7fc2f006ea40 0x7fc2f006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2efe00640 1 -- 172.21.15.90:0/2172596947 >> v2:172.21.15.102:3300/0 conn(0x7fc2f006ea40 msgr2=0x7fc2f006eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:09.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2efe00640 1 --2- 172.21.15.90:0/2172596947 >> v2:172.21.15.102:3300/0 conn(0x7fc2f006ea40 0x7fc2f006eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2efe00640 1 -- 172.21.15.90:0/2172596947 >> v1:172.21.15.102:6789/0 conn(0x7fc2f00d00c0 legacy=0x7fc2f006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2efe00640 1 -- 172.21.15.90:0/2172596947 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f00ce7c0 con 0x7fc2f015e370 2024-03-31T15:56:09.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2efe00640 1 --2- 172.21.15.90:0/2172596947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f006cba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00b00e0 tx=0x7fc2e001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2271465b6bbf8dee server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2ef400640 1 -- 172.21.15.90:0/2172596947 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00b2070 con 0x7fc2f015e370 2024-03-31T15:56:09.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2ef400640 1 -- 172.21.15.90:0/2172596947 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e00ad680 con 0x7fc2f015e370 2024-03-31T15:56:09.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2ef400640 1 -- 172.21.15.90:0/2172596947 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00ada50 con 0x7fc2f015e370 2024-03-31T15:56:09.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2f7200640 1 -- 172.21.15.90:0/2172596947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 msgr2=0x7fc2f006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.040+0000 7fc2f7200640 1 --2- 172.21.15.90:0/2172596947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f006cba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fc2f00b00e0 tx=0x7fc2e001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/2172596947 shutdown_connections 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 --2- 172.21.15.90:0/2172596947 >> v2:172.21.15.102:3300/0 conn(0x7fc2f006ea40 0x7fc2f006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 --2- 172.21.15.90:0/2172596947 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f006cba0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/2172596947 >> 172.21.15.90:0/2172596947 conn(0x7fc2f00d0890 msgr2=0x7fc2f00d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/2172596947 shutdown_connections 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/2172596947 wait complete. 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 Processor -- start 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- start start 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f006ea40 0x7fc2f01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f00d00c0 0x7fc2f0157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0162890 con 0x7fc2f015e370 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f0162710 con 0x7fc2f006ea40 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f0162a10 con 0x7fc2f00d00c0 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:52998/0 (socket says 172.21.15.90:52998) 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 -- 172.21.15.90:0/1870596916 learned_addr learned my addr 172.21.15.90:0/1870596916 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2efe00640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f006ea40 0x7fc2f01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2e7e00640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f00d00c0 0x7fc2f0157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 -- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f00d00c0 msgr2=0x7fc2f0157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f00d00c0 0x7fc2f0157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 -- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f006ea40 msgr2=0x7fc2f01576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f006ea40 0x7fc2f01576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 -- 172.21.15.90:0/1870596916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f00c54c0 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f5200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800efc0 tx=0x7fc2e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8018020 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8014480 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f00c57a0 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8014870 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.044+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f00c5d60 con 0x7fc2f015e370 2024-03-31T15:56:09.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f00685f0 con 0x7fc2f015e370 2024-03-31T15:56:09.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc2e801b020 con 0x7fc2f015e370 2024-03-31T15:56:09.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2ecc00640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 0x7fc2cc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fc2e80876b0 con 0x7fc2f015e370 2024-03-31T15:56:09.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2efe00640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 0x7fc2cc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.048+0000 7fc2efe00640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 0x7fc2cc06a870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00af000 tx=0x7fc2e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.052+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc2e8022070 con 0x7fc2f015e370 2024-03-31T15:56:09.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.264+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7fc2f00d0e80 con 0x7fc2f015e370 2024-03-31T15:56:09.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.268+0000 7fc2ecc00640 1 -- 172.21.15.90:0/1870596916 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) ==== 184+0+141 (secure 0 0 0) 0x7fc2e8010040 con 0x7fc2f015e370 2024-03-31T15:56:09.262 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:56:09.263 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:56:09.263 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:56:09.263 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:56:09.263 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-31T15:56:09.263 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 msgr2=0x7fc2cc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 0x7fc2cc06a870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00af000 tx=0x7fc2e00a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 msgr2=0x7fc2f015a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800efc0 tx=0x7fc2e800c490 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 shutdown_connections 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc2cc0683b0 0x7fc2cc06a870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2f015e370 0x7fc2f015a930 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc2f00d00c0 0x7fc2f0157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 --2- 172.21.15.90:0/1870596916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2f006ea40 0x7fc2f01576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 >> 172.21.15.90:0/1870596916 conn(0x7fc2f00d0890 msgr2=0x7fc2f0160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 shutdown_connections 2024-03-31T15:56:09.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.272+0000 7fc2f7200640 1 -- 172.21.15.90:0/1870596916 wait complete. 2024-03-31T15:56:09.284 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:56:09.284 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:09.374 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:09.468 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:56:09.468 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:09.468 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:09.468 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:09.478 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:56:09.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:09.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:09.478 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:56:09.528 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-31T15:56:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:09.529 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:09.529 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:09.529 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:09.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:09.538 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:56:09.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:09.538 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:56:09.588 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:56:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:09.590 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:09.590 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:09.590 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:09.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:56:09.599 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:09.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:09.599 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:56:09.653 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:56:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:09.654 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:09.654 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:09.654 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:09.663 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:09.663 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:56:09.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:09.663 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:56:09.712 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:56:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:09.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.856+0000 7ff161c00640 1 Processor -- start 2024-03-31T15:56:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.856+0000 7ff161c00640 1 -- start start 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff161c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff15c15e4c0 0x7ff15c0d41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff161c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff15c06c990 0x7ff15c0cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff161c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c15e8a0 con 0x7ff15c0d4700 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff161c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff15c06cf90 con 0x7ff15c06c990 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff161c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff15c06d270 con 0x7ff15c15e4c0 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff15c06c990 0x7ff15c0cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff15c06c990 0x7ff15c0cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53000/0 (socket says 172.21.15.90:53000) 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff15c15e4c0 0x7ff15c0d41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 -- 172.21.15.90:0/3723740857 learned_addr learned my addr 172.21.15.90:0/3723740857 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 -- 172.21.15.90:0/3723740857 >> v2:172.21.15.102:3300/0 conn(0x7ff15c15e4c0 msgr2=0x7ff15c0d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 --2- 172.21.15.90:0/3723740857 >> v2:172.21.15.102:3300/0 conn(0x7ff15c15e4c0 0x7ff15c0d41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 -- 172.21.15.90:0/3723740857 >> v1:172.21.15.102:6789/0 conn(0x7ff15c0d4700 legacy=0x7ff15c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 -- 172.21.15.90:0/3723740857 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c06d550 con 0x7ff15c06c990 2024-03-31T15:56:09.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15a000640 1 -- 172.21.15.90:0/3723740857 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279226685 0 0) 0x7ff15c15e8a0 con 0x7ff15c0d4700 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.860+0000 7ff15b400640 1 --2- 172.21.15.90:0/3723740857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c06c990 0x7ff15c0cf480 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0d1ce0 tx=0x7ff15401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491ff7b7ff0214cb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15a000640 1 -- 172.21.15.90:0/3723740857 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540a56c0 con 0x7ff15c06c990 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15a000640 1 -- 172.21.15.90:0/3723740857 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1540a5860 con 0x7ff15c06c990 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15a000640 1 -- 172.21.15.90:0/3723740857 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1540a5c30 con 0x7ff15c06c990 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- 172.21.15.90:0/3723740857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c06c990 msgr2=0x7ff15c0cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- 172.21.15.90:0/3723740857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c06c990 0x7ff15c0cf480 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0d1ce0 tx=0x7ff15401c9f0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- 172.21.15.90:0/3723740857 shutdown_connections 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- 172.21.15.90:0/3723740857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c06c990 0x7ff15c0cf480 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- 172.21.15.90:0/3723740857 >> v2:172.21.15.102:3300/0 conn(0x7ff15c15e4c0 0x7ff15c0d41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- 172.21.15.90:0/3723740857 >> 172.21.15.90:0/3723740857 conn(0x7ff15c08db60 msgr2=0x7ff15c09d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- 172.21.15.90:0/3723740857 shutdown_connections 2024-03-31T15:56:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- 172.21.15.90:0/3723740857 wait complete. 2024-03-31T15:56:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 Processor -- start 2024-03-31T15:56:09.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- start start 2024-03-31T15:56:09.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 0x7ff15c14b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff15c0d4700 0x7ff15c153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c0d05a0 con 0x7ff15c15e4c0 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c0d0420 con 0x7ff15c06c990 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff161c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff15c0d0720 con 0x7ff15c0d4700 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53006/0 (socket says 172.21.15.90:53006) 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15b400640 1 -- 172.21.15.90:0/1005539740 learned_addr learned my addr 172.21.15.90:0/1005539740 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff15aa00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 0x7ff15c14b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.864+0000 7ff153e00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff15c0d4700 0x7ff15c153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 -- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff15c0d4700 msgr2=0x7ff15c153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff15c0d4700 0x7ff15c153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 -- 172.21.15.90:0/1005539740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 msgr2=0x7ff15c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 -- 172.21.15.90:0/1005539740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c1502a0 con 0x7ff15c06c990 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15b400640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff15aa00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 0x7ff15c14b490 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00e870 tx=0x7ff14c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c017070 con 0x7ff15c06c990 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff14c00ee60 con 0x7ff15c06c990 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c150580 con 0x7ff15c06c990 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c00cb60 con 0x7ff15c06c990 2024-03-31T15:56:09.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c150b40 con 0x7ff15c06c990 2024-03-31T15:56:09.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15c0686d0 con 0x7ff15c06c990 2024-03-31T15:56:09.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff14c01a070 con 0x7ff15c06c990 2024-03-31T15:56:09.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 0x7ff13806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:09.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.868+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ff14c086c80 con 0x7ff15c06c990 2024-03-31T15:56:09.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.872+0000 7ff153e00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 0x7ff13806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:09.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.872+0000 7ff153e00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 0x7ff13806a840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff15c15b2e0 tx=0x7ff144009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:09.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:09.876+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff14c056790 con 0x7ff15c06c990 2024-03-31T15:56:10.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.076+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff15c0b6b10 con 0x7ff15c06c990 2024-03-31T15:56:10.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.076+0000 7ff152a00640 1 -- 172.21.15.90:0/1005539740 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5186 (secure 0 0 0) 0x7ff14c05aa90 con 0x7ff15c06c990 2024-03-31T15:56:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:10.070 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:31.990711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:56:10.070 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-03-31T15:56:10.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 msgr2=0x7ff13806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 0x7ff13806a840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff15c15b2e0 tx=0x7ff144009040 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 msgr2=0x7ff15c14b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 0x7ff15c14b490 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00e870 tx=0x7ff14c00ed30 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 shutdown_connections 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff138068380 0x7ff13806a840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff15c15e4c0 0x7ff15c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff15c0d4700 0x7ff15c153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 --2- 172.21.15.90:0/1005539740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff15c06c990 0x7ff15c14b490 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 >> 172.21.15.90:0/1005539740 conn(0x7ff15c08db60 msgr2=0x7ff15c15ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 shutdown_connections 2024-03-31T15:56:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.080+0000 7ff161c00640 1 -- 172.21.15.90:0/1005539740 wait complete. 2024-03-31T15:56:10.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.240+0000 7f19a9600640 1 Processor -- start 2024-03-31T15:56:10.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.240+0000 7f19a9600640 1 -- start start 2024-03-31T15:56:10.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a9600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19a415c120 0x7f19a415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a9600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a9600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19a4160350 con 0x7f19a40b2c20 2024-03-31T15:56:10.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a9600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f19a4160bf0 con 0x7f19a415c120 2024-03-31T15:56:10.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a9600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f19a4160630 con 0x7f19a40d2680 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19a415c120 0x7f19a415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:55388/0 (socket says 172.21.15.90:55388) 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19a415c120 0x7f19a415fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:56374/0 (socket says 172.21.15.90:56374) 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2000640 1 -- 172.21.15.90:0/2007409898 learned_addr learned my addr 172.21.15.90:0/2007409898 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a3400640 1 --1- 172.21.15.90:0/2007409898 >> v1:172.21.15.102:6789/0 conn(0x7f19a40b2c20 0x7f19a40cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:34562/0 (socket says 172.21.15.90:34562) 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 -- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 msgr2=0x7f19a40b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 -- 172.21.15.90:0/2007409898 >> v1:172.21.15.102:6789/0 conn(0x7f19a40b2c20 legacy=0x7f19a40cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2a00640 1 -- 172.21.15.90:0/2007409898 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a4160910 con 0x7f19a415c120 2024-03-31T15:56:10.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a2000640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:10.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.244+0000 7f19a1600640 1 -- 172.21.15.90:0/2007409898 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309219377 0 0) 0x7f19a4160350 con 0x7f19a40b2c20 2024-03-31T15:56:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a2a00640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415fd00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f19940097c0 tx=0x7f199401c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d580cac21eff4d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a1600640 1 -- 172.21.15.90:0/2007409898 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940b0070 con 0x7f19a415c120 2024-03-31T15:56:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a1600640 1 -- 172.21.15.90:0/2007409898 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19940ab600 con 0x7f19a415c120 2024-03-31T15:56:10.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a1600640 1 -- 172.21.15.90:0/2007409898 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19940ab9d0 con 0x7f19a415c120 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 -- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 msgr2=0x7f19a415fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415fd00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f19940097c0 tx=0x7f199401c350 comp rx=0 tx=0).stop 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 -- 172.21.15.90:0/2007409898 shutdown_connections 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40d2680 0x7f19a40b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 --2- 172.21.15.90:0/2007409898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415fd00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 -- 172.21.15.90:0/2007409898 >> 172.21.15.90:0/2007409898 conn(0x7f19a409aa40 msgr2=0x7f19a409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 -- 172.21.15.90:0/2007409898 shutdown_connections 2024-03-31T15:56:10.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.248+0000 7f19a9600640 1 -- 172.21.15.90:0/2007409898 wait complete. 2024-03-31T15:56:10.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 Processor -- start 2024-03-31T15:56:10.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- start start 2024-03-31T15:56:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40b2c20 0x7f19a40cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19a41691c0 con 0x7f19a40d2680 2024-03-31T15:56:10.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4169040 con 0x7f19a415c120 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f19a4169340 con 0x7f19a40b2c20 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53014/0 (socket says 172.21.15.90:53014) 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 -- 172.21.15.90:0/477078224 learned_addr learned my addr 172.21.15.90:0/477078224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a3400640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2a00640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40b2c20 0x7f19a40cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 -- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40b2c20 msgr2=0x7f19a40cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40b2c20 0x7f19a40cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 -- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 msgr2=0x7f19a415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 -- 172.21.15.90:0/477078224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a414baa0 con 0x7f19a40d2680 2024-03-31T15:56:10.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a2000640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f199000e960 tx=0x7f199000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1990017070 con 0x7f19a40d2680 2024-03-31T15:56:10.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f199000ca00 con 0x7f19a40d2680 2024-03-31T15:56:10.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a40ca970 con 0x7f19a40d2680 2024-03-31T15:56:10.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199000cdb0 con 0x7f19a40d2680 2024-03-31T15:56:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.252+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a40caf30 con 0x7f19a40d2680 2024-03-31T15:56:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a40686d0 con 0x7f19a40d2680 2024-03-31T15:56:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f199001a070 con 0x7f19a40d2680 2024-03-31T15:56:10.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f199aa00640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 0x7f198406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f1990086c20 con 0x7f19a40d2680 2024-03-31T15:56:10.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f19a2a00640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 0x7f198406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.256+0000 7f19a2a00640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 0x7f198406a840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f19940ad000 tx=0x7f19940a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.260+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1990056730 con 0x7f19a40d2680 2024-03-31T15:56:10.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.416+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19a40b1b60 con 0x7f19a40d2680 2024-03-31T15:56:10.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.416+0000 7f199aa00640 1 -- 172.21.15.90:0/477078224 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f199005aa30 con 0x7f19a40d2680 2024-03-31T15:56:10.412 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:10.412 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 msgr2=0x7f198406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 0x7f198406a840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f19940ad000 tx=0x7f19940a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 msgr2=0x7f19a4149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f199000e960 tx=0x7f199000ee20 comp rx=0 tx=0).stop 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 shutdown_connections 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1984068380 0x7f198406a840 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19a415c120 0x7f19a415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19a40d2680 0x7f19a4149a10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 --2- 172.21.15.90:0/477078224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19a40b2c20 0x7f19a40cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 >> 172.21.15.90:0/477078224 conn(0x7f19a409aa40 msgr2=0x7f19a406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 shutdown_connections 2024-03-31T15:56:10.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.420+0000 7f19a9600640 1 -- 172.21.15.90:0/477078224 wait complete. 2024-03-31T15:56:10.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:55:15.583234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:55:16.891180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-31T15:55:31.097739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:10.435 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 Processor -- start 2024-03-31T15:56:10.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 -- start start 2024-03-31T15:56:10.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc415e4e0 0x7f2fc406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc406ce50 0x7f2fc406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fc415e8c0 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc40d0ff0 con 0x7f2fc406ce50 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc9000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc40d12d0 con 0x7f2fc415e4e0 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc406ce50 0x7f2fc406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2fc406ce50 0x7f2fc406d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53016/0 (socket says 172.21.15.90:53016) 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 -- 172.21.15.90:0/2811846776 learned_addr learned my addr 172.21.15.90:0/2811846776 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1e00640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc415e4e0 0x7f2fc406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 -- 172.21.15.90:0/2811846776 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc415e4e0 msgr2=0x7f2fc406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc415e4e0 0x7f2fc406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 -- 172.21.15.90:0/2811846776 >> v1:172.21.15.102:6789/0 conn(0x7f2fc40ce7a0 legacy=0x7f2fc40cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.580+0000 7f2fc1400640 1 -- 172.21.15.90:0/2811846776 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40cc710 con 0x7f2fc406ce50 2024-03-31T15:56:10.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc1400640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc406ce50 0x7f2fc406d290 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8274a1a965b149 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc0a00640 1 -- 172.21.15.90:0/2811846776 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb00b0070 con 0x7f2fc406ce50 2024-03-31T15:56:10.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc0a00640 1 -- 172.21.15.90:0/2811846776 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb00ab600 con 0x7f2fc406ce50 2024-03-31T15:56:10.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc0a00640 1 -- 172.21.15.90:0/2811846776 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb00ab9d0 con 0x7f2fc406ce50 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 -- 172.21.15.90:0/2811846776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc406ce50 msgr2=0x7f2fc406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc406ce50 0x7f2fc406d290 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009870 tx=0x7f2fb001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 -- 172.21.15.90:0/2811846776 shutdown_connections 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc406ce50 0x7f2fc406d290 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 --2- 172.21.15.90:0/2811846776 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc415e4e0 0x7f2fc406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 -- 172.21.15.90:0/2811846776 >> 172.21.15.90:0/2811846776 conn(0x7f2fc40d1f20 msgr2=0x7f2fc40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 -- 172.21.15.90:0/2811846776 shutdown_connections 2024-03-31T15:56:10.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.584+0000 7f2fc9000640 1 -- 172.21.15.90:0/2811846776 wait complete. 2024-03-31T15:56:10.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 Processor -- start 2024-03-31T15:56:10.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- start start 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc406ce50 0x7f2fc41544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc415e4e0 0x7f2fc40c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40d0d90 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40d0c10 con 0x7f2fc415e4e0 2024-03-31T15:56:10.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc416b840 con 0x7f2fc406ce50 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53020/0 (socket says 172.21.15.90:53020) 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 -- 172.21.15.90:0/3455115423 learned_addr learned my addr 172.21.15.90:0/3455115423 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1e00640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc406ce50 0x7f2fc41544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc2800640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc415e4e0 0x7f2fc40c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 -- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc406ce50 msgr2=0x7f2fc41544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc406ce50 0x7f2fc41544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 -- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc415e4e0 msgr2=0x7f2fc40c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc415e4e0 0x7f2fc40c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 -- 172.21.15.90:0/3455115423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40cb790 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc1400640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00ae040 tx=0x7f2fb0019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb00ba070 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb00b4430 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc40c8120 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb00b0040 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.588+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc40cba30 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.592+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc40685f0 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.592+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2fb00bd070 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.592+0000 7f2fba000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 0x7f2fa406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.592+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f2fb011ff10 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.592+0000 7f2fc1e00640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 0x7f2fa406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.596+0000 7f2fc1e00640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 0x7f2fa406a840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40b7cf0 tx=0x7f2fb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.600+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2fb0016150 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.756+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2fc40b2c60 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.756+0000 7f2fba000640 1 -- 172.21.15.90:0/3455115423 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7f2fb00efbe0 con 0x7f2fc40ce7a0 2024-03-31T15:56:10.752 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 msgr2=0x7f2fa406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 0x7f2fa406a840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40b7cf0 tx=0x7f2fb4009190 comp rx=0 tx=0).stop 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 msgr2=0x7f2fc4152740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00ae040 tx=0x7f2fb0019270 comp rx=0 tx=0).stop 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 shutdown_connections 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2fa4068380 0x7f2fa406a840 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fc415e4e0 0x7f2fc40c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fc40ce7a0 0x7f2fc4152740 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 --2- 172.21.15.90:0/3455115423 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2fc406ce50 0x7f2fc41544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 >> 172.21.15.90:0/3455115423 conn(0x7f2fc40d1f20 msgr2=0x7f2fc406d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 shutdown_connections 2024-03-31T15:56:10.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.760+0000 7f2fc9000640 1 -- 172.21.15.90:0/3455115423 wait complete. 2024-03-31T15:56:10.773 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:10.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 Processor -- start 2024-03-31T15:56:10.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 -- start start 2024-03-31T15:56:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5bd815c320 0x7f5bd80cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5bd80d2b20 0x7f5bd80ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 0x7f5bd80cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5bd80aed70 con 0x7f5bd80d2b20 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5bd80cd900 con 0x7f5bd815c320 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bde800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd80cefc0 con 0x7f5bd80cd470 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5bd80d2b20 0x7f5bd80ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5bd80d2b20 0x7f5bd80ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53036/0 (socket says 172.21.15.90:53036) 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 -- 172.21.15.90:0/844333600 learned_addr learned my addr 172.21.15.90:0/844333600 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bd7e00640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 0x7f5bd80cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bd7400640 1 --2- 172.21.15.90:0/844333600 >> v2:172.21.15.102:3300/0 conn(0x7f5bd815c320 0x7f5bd80cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 -- 172.21.15.90:0/844333600 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 msgr2=0x7f5bd80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 0x7f5bd80cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 -- 172.21.15.90:0/844333600 >> v2:172.21.15.102:3300/0 conn(0x7f5bd815c320 msgr2=0x7f5bd80cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/844333600 >> v2:172.21.15.102:3300/0 conn(0x7f5bd815c320 0x7f5bd80cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bcfe00640 1 -- 172.21.15.90:0/844333600 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd80cf2a0 con 0x7f5bd80d2b20 2024-03-31T15:56:10.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.920+0000 7f5bd7e00640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 0x7f5bd80cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80d2b20 0x7f5bd80ccf30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80739b0 tx=0x7f5bc001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c906bb7290a00c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bd6a00640 1 -- 172.21.15.90:0/844333600 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00b0070 con 0x7f5bd80d2b20 2024-03-31T15:56:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bd6a00640 1 -- 172.21.15.90:0/844333600 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc00ab600 con 0x7f5bd80d2b20 2024-03-31T15:56:10.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bd6a00640 1 -- 172.21.15.90:0/844333600 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00ab9d0 con 0x7f5bd80d2b20 2024-03-31T15:56:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 -- 172.21.15.90:0/844333600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80d2b20 msgr2=0x7f5bd80ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80d2b20 0x7f5bd80ccf30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80739b0 tx=0x7f5bc001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:10.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 -- 172.21.15.90:0/844333600 shutdown_connections 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd80cd470 0x7f5bd80cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 --2- 172.21.15.90:0/844333600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80d2b20 0x7f5bd80ccf30 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 --2- 172.21.15.90:0/844333600 >> v2:172.21.15.102:3300/0 conn(0x7f5bd815c320 0x7f5bd80cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 -- 172.21.15.90:0/844333600 >> 172.21.15.90:0/844333600 conn(0x7f5bd80d1f20 msgr2=0x7f5bd80d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 -- 172.21.15.90:0/844333600 shutdown_connections 2024-03-31T15:56:10.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.924+0000 7f5bde800640 1 -- 172.21.15.90:0/844333600 wait complete. 2024-03-31T15:56:10.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 Processor -- start 2024-03-31T15:56:10.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- start start 2024-03-31T15:56:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bd80d2b20 0x7f5bd8156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd815c320 0x7f5bd8165440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd80d0e20 con 0x7f5bd80cd470 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd80d0ca0 con 0x7f5bd80d2b20 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd80af1e0 con 0x7f5bd815c320 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53042/0 (socket says 172.21.15.90:53042) 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 -- 172.21.15.90:0/2244599557 learned_addr learned my addr 172.21.15.90:0/2244599557 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bd80d2b20 0x7f5bd8156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7e00640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd815c320 0x7f5bd8165440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 -- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd815c320 msgr2=0x7f5bd8165440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd815c320 0x7f5bd8165440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 -- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bd80d2b20 msgr2=0x7f5bd8156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bd80d2b20 0x7f5bd8156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 -- 172.21.15.90:0/2244599557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd806dcd0 con 0x7f5bd80cd470 2024-03-31T15:56:10.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bd7400640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800d5f0 tx=0x7f5bc800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80125f0 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc8019070 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80129b0 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd806dfb0 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd806e680 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.928+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd80685f0 con 0x7f5bd80cd470 2024-03-31T15:56:10.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.932+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5bc8012b50 con 0x7f5bd80cd470 2024-03-31T15:56:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.932+0000 7f5bcf400640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 0x7f5bb406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.932+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f5bc8086fe0 con 0x7f5bd80cd470 2024-03-31T15:56:10.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.932+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 0x7f5bb406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:10.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.932+0000 7f5bcfe00640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 0x7f5bb406a840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00ad000 tx=0x7f5bc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:10.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:10.936+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5bc8057260 con 0x7f5bd80cd470 2024-03-31T15:56:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.092+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5bd80b0050 con 0x7f5bd80cd470 2024-03-31T15:56:11.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.092+0000 7f5bcf400640 1 -- 172.21.15.90:0/2244599557 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f5bc8056b60 con 0x7f5bd80cd470 2024-03-31T15:56:11.087 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:11.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 msgr2=0x7f5bb406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 0x7f5bb406a840 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00ad000 tx=0x7f5bc0008040 comp rx=0 tx=0).stop 2024-03-31T15:56:11.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 msgr2=0x7f5bd8150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800d5f0 tx=0x7f5bc800dab0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 shutdown_connections 2024-03-31T15:56:11.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bb4068380 0x7f5bb406a840 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5bd815c320 0x7f5bd8165440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bd80d2b20 0x7f5bd8156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 --2- 172.21.15.90:0/2244599557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5bd80cd470 0x7f5bd8150140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 >> 172.21.15.90:0/2244599557 conn(0x7f5bd80d1f20 msgr2=0x7f5bd80cca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 shutdown_connections 2024-03-31T15:56:11.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.096+0000 7f5bde800640 1 -- 172.21.15.90:0/2244599557 wait complete. 2024-03-31T15:56:11.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:11.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.268+0000 7fad48200640 1 Processor -- start 2024-03-31T15:56:11.263 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.268+0000 7fad48200640 1 -- start start 2024-03-31T15:56:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad4015c0b0 0x7fad400afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad400b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad400b3240 0x7fad4006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fad4015c490 con 0x7fad400b3240 2024-03-31T15:56:11.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fad400b3670 con 0x7fad4015c0b0 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad48200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fad4015fef0 con 0x7fad400ccfe0 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad400b3240 0x7fad4006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fad400b3240 0x7fad4006b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53048/0 (socket says 172.21.15.90:53048) 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 -- 172.21.15.90:0/3822279915 learned_addr learned my addr 172.21.15.90:0/3822279915 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad45800640 1 --2- 172.21.15.90:0/3822279915 >> v2:172.21.15.102:3300/0 conn(0x7fad4015c0b0 0x7fad400afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad44e00640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad400b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 -- 172.21.15.90:0/3822279915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 msgr2=0x7fad400b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad400b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.265 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 -- 172.21.15.90:0/3822279915 >> v2:172.21.15.102:3300/0 conn(0x7fad4015c0b0 msgr2=0x7fad400afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 --2- 172.21.15.90:0/3822279915 >> v2:172.21.15.102:3300/0 conn(0x7fad4015c0b0 0x7fad400afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 -- 172.21.15.90:0/3822279915 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad401601d0 con 0x7fad400b3240 2024-03-31T15:56:11.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.272+0000 7fad46200640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad400b3240 0x7fad4006b5f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fad400723e0 tx=0x7fad3801c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e1e99621d6fa29 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad3fe00640 1 -- 172.21.15.90:0/3822279915 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380a5580 con 0x7fad400b3240 2024-03-31T15:56:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad3fe00640 1 -- 172.21.15.90:0/3822279915 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad380a5720 con 0x7fad400b3240 2024-03-31T15:56:11.268 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad3fe00640 1 -- 172.21.15.90:0/3822279915 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380a5af0 con 0x7fad400b3240 2024-03-31T15:56:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- 172.21.15.90:0/3822279915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad400b3240 msgr2=0x7fad4006b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad400b3240 0x7fad4006b5f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fad400723e0 tx=0x7fad3801c700 comp rx=0 tx=0).stop 2024-03-31T15:56:11.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- 172.21.15.90:0/3822279915 shutdown_connections 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad400b3240 0x7fad4006b5f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- 172.21.15.90:0/3822279915 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad400b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- 172.21.15.90:0/3822279915 >> v2:172.21.15.102:3300/0 conn(0x7fad4015c0b0 0x7fad400afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- 172.21.15.90:0/3822279915 >> 172.21.15.90:0/3822279915 conn(0x7fad400cde90 msgr2=0x7fad400ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- 172.21.15.90:0/3822279915 shutdown_connections 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- 172.21.15.90:0/3822279915 wait complete. 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 Processor -- start 2024-03-31T15:56:11.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- start start 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad400b3240 0x7fad40155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad40157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fad4015fba0 con 0x7fad4015c0b0 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fad40160a70 con 0x7fad400b3240 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad48200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fad400cb700 con 0x7fad400ccfe0 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad46200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad46200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53062/0 (socket says 172.21.15.90:53062) 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad46200640 1 -- 172.21.15.90:0/819309412 learned_addr learned my addr 172.21.15.90:0/819309412 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad44e00640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad40157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.276+0000 7fad45800640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad400b3240 0x7fad40155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 -- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 msgr2=0x7fad40157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad40157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 -- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad400b3240 msgr2=0x7fad40155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad400b3240 0x7fad40155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 -- 172.21.15.90:0/819309412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad401545d0 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad46200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fad380af040 tx=0x7fad380194a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38009510 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad380be070 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad40156360 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380b88d0 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad40156920 con 0x7fad4015c0b0 2024-03-31T15:56:11.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad400685f0 con 0x7fad4015c0b0 2024-03-31T15:56:11.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fad380b8a70 con 0x7fad4015c0b0 2024-03-31T15:56:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 0x7fad2006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.280+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fad38121520 con 0x7fad4015c0b0 2024-03-31T15:56:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.284+0000 7fad45800640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 0x7fad2006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.284+0000 7fad45800640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 0x7fad2006a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad400d0bc0 tx=0x7fad34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.284+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fad380c7020 con 0x7fad4015c0b0 2024-03-31T15:56:11.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.440+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fad400ccb20 con 0x7fad4015c0b0 2024-03-31T15:56:11.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad3d600640 1 -- 172.21.15.90:0/819309412 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7fad380165e0 con 0x7fad4015c0b0 2024-03-31T15:56:11.436 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 msgr2=0x7fad2006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 0x7fad2006a870 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad400d0bc0 tx=0x7fad34008040 comp rx=0 tx=0).stop 2024-03-31T15:56:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 msgr2=0x7fad400cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fad380af040 tx=0x7fad380194a0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 shutdown_connections 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fad200683b0 0x7fad2006a870 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fad4015c0b0 0x7fad400cfc70 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fad400ccfe0 0x7fad40157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 --2- 172.21.15.90:0/819309412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad400b3240 0x7fad40155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 >> 172.21.15.90:0/819309412 conn(0x7fad400cde90 msgr2=0x7fad4006ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 shutdown_connections 2024-03-31T15:56:11.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.444+0000 7fad48200640 1 -- 172.21.15.90:0/819309412 wait complete. 2024-03-31T15:56:11.458 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-31T15:56:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 Processor -- start 2024-03-31T15:56:11.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 -- start start 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbc7c15e4e0 0x7fbc7c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c0cdf20 0x7fbc7c06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7c15e8c0 con 0x7fbc7c06ce50 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbc7c0d0ff0 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc80e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c0d12d0 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc79e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbc7c15e4e0 0x7fbc7c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc73e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbc7c06ce50 0x7fbc7c0ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43192/0 (socket says 172.21.15.90:43192) 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc73e00640 1 -- 172.21.15.90:0/2004907463 learned_addr learned my addr 172.21.15.90:0/2004907463 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc7a800640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c0cdf20 0x7fbc7c06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc79e00640 1 -- 172.21.15.90:0/2004907463 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c0cdf20 msgr2=0x7fbc7c06d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc79e00640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c0cdf20 0x7fbc7c06d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc79e00640 1 -- 172.21.15.90:0/2004907463 >> v1:172.21.15.90:6789/0 conn(0x7fbc7c06ce50 legacy=0x7fbc7c0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc79e00640 1 -- 172.21.15.90:0/2004907463 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c0cc710 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.604+0000 7fbc79400640 1 -- 172.21.15.90:0/2004907463 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913120510 0 0) 0x7fbc7c15e8c0 con 0x7fbc7c06ce50 2024-03-31T15:56:11.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc79e00640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c06c910 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0097f0 tx=0x7fbc6c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc9bf7a1950df5f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc79400640 1 -- 172.21.15.90:0/2004907463 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0b0070 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc79400640 1 -- 172.21.15.90:0/2004907463 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c0ab600 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc79400640 1 -- 172.21.15.90:0/2004907463 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0ab9d0 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 -- 172.21.15.90:0/2004907463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c15e4e0 msgr2=0x7fbc7c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c06c910 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0097f0 tx=0x7fbc6c01c410 comp rx=0 tx=0).stop 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 -- 172.21.15.90:0/2004907463 shutdown_connections 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c0cdf20 0x7fbc7c06d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 --2- 172.21.15.90:0/2004907463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c06c910 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 -- 172.21.15.90:0/2004907463 >> 172.21.15.90:0/2004907463 conn(0x7fbc7c0d1f20 msgr2=0x7fbc7c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 -- 172.21.15.90:0/2004907463 shutdown_connections 2024-03-31T15:56:11.604 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.608+0000 7fbc80e00640 1 -- 172.21.15.90:0/2004907463 wait complete. 2024-03-31T15:56:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 Processor -- start 2024-03-31T15:56:11.605 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- start start 2024-03-31T15:56:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c06ce50 0x7fbc7c06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c16b1b0 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c16b030 con 0x7fbc7c15e4e0 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c16b330 con 0x7fbc7c06ce50 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53084/0 (socket says 172.21.15.90:53084) 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 -- 172.21.15.90:0/4284849296 learned_addr learned my addr 172.21.15.90:0/4284849296 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc79e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c06ce50 0x7fbc7c06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc7a800640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 -- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c06ce50 msgr2=0x7fbc7c06f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c06ce50 0x7fbc7c06f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 -- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c15e4e0 msgr2=0x7fbc7c158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 -- 172.21.15.90:0/4284849296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c158960 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc73e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbc640011f0 tx=0x7fbc6400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64018070 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc640074e0 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c158c40 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64004a10 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.612+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c159200 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c0685f0 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbc6401b070 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc72000640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 0x7fbc5806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fbc64085f70 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc79e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 0x7fbc5806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.616+0000 7fbc79e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 0x7fbc5806a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0ad000 tx=0x7fbc6c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.620+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbc64056290 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.776+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fbc7c075060 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.776+0000 7fbc72000640 1 -- 172.21.15.90:0/4284849296 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7fbc64055b90 con 0x7fbc7c0cdf20 2024-03-31T15:56:11.772 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 msgr2=0x7fbc5806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 0x7fbc5806a870 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0ad000 tx=0x7fbc6c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 msgr2=0x7fbc7c157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbc640011f0 tx=0x7fbc6400c9b0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 shutdown_connections 2024-03-31T15:56:11.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbc580683b0 0x7fbc5806a870 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c15e4e0 0x7fbc7c158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbc7c0cdf20 0x7fbc7c157d50 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 --2- 172.21.15.90:0/4284849296 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbc7c06ce50 0x7fbc7c06f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 >> 172.21.15.90:0/4284849296 conn(0x7fbc7c0d1f20 msgr2=0x7fbc7c0d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 shutdown_connections 2024-03-31T15:56:11.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.780+0000 7fbc80e00640 1 -- 172.21.15.90:0/4284849296 wait complete. 2024-03-31T15:56:11.795 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.936+0000 7f3b04200640 1 Processor -- start 2024-03-31T15:56:11.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.936+0000 7f3b04200640 1 -- start start 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b04200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3afc15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b04200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3afc0d2680 0x7f3afc0b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b04200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3afc160350 con 0x7f3afc0b2c20 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b04200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3afc160bf0 con 0x7f3afc0d2680 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b04200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc160630 con 0x7f3afc15c120 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b02200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3afc0b2c20 0x7f3afc0cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43206/0 (socket says 172.21.15.90:43206) 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b02200640 1 -- 172.21.15.90:0/1197439 learned_addr learned my addr 172.21.15.90:0/1197439 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b00e00640 1 --2- 172.21.15.90:0/1197439 >> v2:172.21.15.102:3300/0 conn(0x7f3afc0d2680 0x7f3afc0b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3afc15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 281806437 0 0) 0x7f3afc160350 con 0x7f3afc0b2c20 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae4003530 con 0x7f3afc0b2c20 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3290758300 0 0) 0x7f3ae4003530 con 0x7f3afc0b2c20 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 msgr2=0x7f3afc15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3afc15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 >> v2:172.21.15.102:3300/0 conn(0x7f3afc0d2680 msgr2=0x7f3afc0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 --2- 172.21.15.90:0/1197439 >> v2:172.21.15.102:3300/0 conn(0x7f3afc0d2680 0x7f3afc0b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc160910 con 0x7f3afc0b2c20 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f3af4001a00 con 0x7f3afc0b2c20 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 >> v1:172.21.15.90:6789/0 conn(0x7f3afc0b2c20 legacy=0x7f3afc0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ae40039c0 0x7f3ae4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3ae4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae4004d50 0x7f3ae40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc160bf0 con 0x7f3ae40039c0 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc160630 con 0x7f3ae4004d50 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae400ba50 con 0x7f3afc15c120 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ae40039c0 0x7f3ae4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b02200640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae4004d50 0x7f3ae40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b00e00640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3ae4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 -- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 msgr2=0x7f3ae4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc15c120 0x7f3ae4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 -- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae4004d50 msgr2=0x7f3ae40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae4004d50 0x7f3ae40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.940+0000 7f3b01800640 1 -- 172.21.15.90:0/1197439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae4003530 con 0x7f3ae40039c0 2024-03-31T15:56:11.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b01800640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ae40039c0 0x7f3ae4003de0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009ae0 tx=0x7f3af001ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ef5aa294940ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af00b2070 con 0x7f3ae40039c0 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af00a5630 con 0x7f3ae40039c0 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3afbe00640 1 -- 172.21.15.90:0/1197439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af00096b0 con 0x7f3ae40039c0 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 -- 172.21.15.90:0/1197439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ae40039c0 msgr2=0x7f3ae4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 --2- 172.21.15.90:0/1197439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ae40039c0 0x7f3ae4003de0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009ae0 tx=0x7f3af001ca40 comp rx=0 tx=0).stop 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b02200640 1 -- 172.21.15.90:0/1197439 reap_dead start 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 -- 172.21.15.90:0/1197439 shutdown_connections 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 -- 172.21.15.90:0/1197439 >> 172.21.15.90:0/1197439 conn(0x7f3afc09aa40 msgr2=0x7f3afc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 -- 172.21.15.90:0/1197439 shutdown_connections 2024-03-31T15:56:11.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.944+0000 7f3b04200640 1 -- 172.21.15.90:0/1197439 wait complete. 2024-03-31T15:56:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 Processor -- start 2024-03-31T15:56:11.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 -- start start 2024-03-31T15:56:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae40039c0 0x7f3afc1f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc0cb520 0x7f3afc153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc0d0cf0 con 0x7f3afc15ec80 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc160110 con 0x7f3ae40039c0 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b04200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc0d0e70 con 0x7f3afc0cb520 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53094/0 (socket says 172.21.15.90:53094) 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 -- 172.21.15.90:0/1297431039 learned_addr learned my addr 172.21.15.90:0/1297431039 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b01800640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae40039c0 0x7f3afc1f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b00e00640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc0cb520 0x7f3afc153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 -- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc0cb520 msgr2=0x7f3afc153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc0cb520 0x7f3afc153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 -- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae40039c0 msgr2=0x7f3afc1f90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae40039c0 0x7f3afc1f90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.948+0000 7f3b02200640 1 -- 172.21.15.90:0/1297431039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc0cbd70 con 0x7f3afc15ec80 2024-03-31T15:56:11.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3b02200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0d2d20 tx=0x7f3af400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af400ebb0 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af400ed50 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4014670 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3afc0b02a0 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3afc15c960 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3afc0686d0 con 0x7f3afc15ec80 2024-03-31T15:56:11.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3af4014810 con 0x7f3afc15ec80 2024-03-31T15:56:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 0x7f3adc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:11.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.952+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f3af4086b00 con 0x7f3afc15ec80 2024-03-31T15:56:11.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.956+0000 7f3b01800640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 0x7f3adc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:11.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.956+0000 7f3b01800640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 0x7f3adc06a840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3af00af000 tx=0x7f3af00046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:11.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:11.956+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3af40566a0 con 0x7f3afc15ec80 2024-03-31T15:56:12.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.148+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3afc0b1b60 con 0x7f3afc15ec80 2024-03-31T15:56:12.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3af9600640 1 -- 172.21.15.90:0/1297431039 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5186 (secure 0 0 0) 0x7f3af4086e20 con 0x7f3afc15ec80 2024-03-31T15:56:12.145 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:12.145 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:31.990711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:56:12.145 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-03-31T15:56:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 msgr2=0x7f3adc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 0x7f3adc06a840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3af00af000 tx=0x7f3af00046d0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 msgr2=0x7f3afc15f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0d2d20 tx=0x7f3af400e530 comp rx=0 tx=0).stop 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 shutdown_connections 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3adc068380 0x7f3adc06a840 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3afc15ec80 0x7f3afc15f100 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3afc0cb520 0x7f3afc153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 --2- 172.21.15.90:0/1297431039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ae40039c0 0x7f3afc1f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 >> 172.21.15.90:0/1297431039 conn(0x7f3afc09aa40 msgr2=0x7f3afc070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 shutdown_connections 2024-03-31T15:56:12.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.152+0000 7f3b04200640 1 -- 172.21.15.90:0/1297431039 wait complete. 2024-03-31T15:56:12.166 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:12.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 Processor -- start 2024-03-31T15:56:12.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 -- start start 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f825015bfd0 0x7f825006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f825006bd60 0x7f825015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82500b47e0 con 0x7f82501449a0 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f825006c1d0 con 0x7f825015bfd0 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f8256600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f82500b4ac0 con 0x7f825006bd60 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f825015bfd0 0x7f825006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f825015bfd0 0x7f825006b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53102/0 (socket says 172.21.15.90:53102) 2024-03-31T15:56:12.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 -- 172.21.15.90:0/1963191086 learned_addr learned my addr 172.21.15.90:0/1963191086 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824fe00640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f825006bd60 0x7f825015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 -- 172.21.15.90:0/1963191086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f825006bd60 msgr2=0x7f825015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f825006bd60 0x7f825015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 -- 172.21.15.90:0/1963191086 >> v1:172.21.15.102:6789/0 conn(0x7f82501449a0 legacy=0x7f82500b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.308+0000 7f824ea00640 1 -- 172.21.15.90:0/1963191086 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825015b7d0 con 0x7f825015bfd0 2024-03-31T15:56:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f824ea00640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f825006b820 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f823c009870 tx=0x7f823c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80592139b5c65651 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f824e000640 1 -- 172.21.15.90:0/1963191086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c0b1070 con 0x7f825015bfd0 2024-03-31T15:56:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f824e000640 1 -- 172.21.15.90:0/1963191086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823c0a8050 con 0x7f825015bfd0 2024-03-31T15:56:12.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f824e000640 1 -- 172.21.15.90:0/1963191086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c0ac860 con 0x7f825015bfd0 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 -- 172.21.15.90:0/1963191086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 msgr2=0x7f825006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f825006b820 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f823c009870 tx=0x7f823c01c560 comp rx=0 tx=0).stop 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 -- 172.21.15.90:0/1963191086 shutdown_connections 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f825006bd60 0x7f825015ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 --2- 172.21.15.90:0/1963191086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f825006b820 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 -- 172.21.15.90:0/1963191086 >> 172.21.15.90:0/1963191086 conn(0x7f82500b1c80 msgr2=0x7f82500b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 -- 172.21.15.90:0/1963191086 shutdown_connections 2024-03-31T15:56:12.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.312+0000 7f8256600640 1 -- 172.21.15.90:0/1963191086 wait complete. 2024-03-31T15:56:12.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 Processor -- start 2024-03-31T15:56:12.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- start start 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825006bd60 0x7f825015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f82501449a0 0x7f82500af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8250167df0 con 0x7f825015bfd0 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f825015b450 con 0x7f825006bd60 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8250167f70 con 0x7f82501449a0 2024-03-31T15:56:12.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53110/0 (socket says 172.21.15.90:53110) 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 -- 172.21.15.90:0/3245693822 learned_addr learned my addr 172.21.15.90:0/3245693822 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824f400640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825006bd60 0x7f825015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824ea00640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f82501449a0 0x7f82500af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 -- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f82501449a0 msgr2=0x7f82500af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f82501449a0 0x7f82500af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 -- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825006bd60 msgr2=0x7f825015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825006bd60 0x7f825015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 -- 172.21.15.90:0/3245693822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825013adb0 con 0x7f825015bfd0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f824fe00640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f824800efc0 tx=0x7f824800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248018020 con 0x7f825015bfd0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8248014480 con 0x7f825015bfd0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825013b090 con 0x7f825015bfd0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248014870 con 0x7f825015bfd0 2024-03-31T15:56:12.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.316+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825013b650 con 0x7f825015bfd0 2024-03-31T15:56:12.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.320+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82500686d0 con 0x7f825015bfd0 2024-03-31T15:56:12.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.320+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f824801b020 con 0x7f825015bfd0 2024-03-31T15:56:12.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.320+0000 7f8247400640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 0x7f823006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.320+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f8248087a80 con 0x7f825015bfd0 2024-03-31T15:56:12.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.320+0000 7f824f400640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 0x7f823006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.324+0000 7f824f400640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 0x7f823006a840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8240001650 tx=0x7f8240009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.328+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8248022030 con 0x7f825015bfd0 2024-03-31T15:56:12.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.484+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8250165bc0 con 0x7f825015bfd0 2024-03-31T15:56:12.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.484+0000 7f8247400640 1 -- 172.21.15.90:0/3245693822 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7f8248010040 con 0x7f825015bfd0 2024-03-31T15:56:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:12.480 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 msgr2=0x7f823006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 0x7f823006a840 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8240001650 tx=0x7f8240009040 comp rx=0 tx=0).stop 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 msgr2=0x7f82500afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f824800efc0 tx=0x7f824800c490 comp rx=0 tx=0).stop 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 shutdown_connections 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8230068380 0x7f823006a840 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f825015bfd0 0x7f82500afdf0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f82501449a0 0x7f82500af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 --2- 172.21.15.90:0/3245693822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f825006bd60 0x7f825015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 >> 172.21.15.90:0/3245693822 conn(0x7f82500b1c80 msgr2=0x7f82500b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 shutdown_connections 2024-03-31T15:56:12.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.488+0000 7f8256600640 1 -- 172.21.15.90:0/3245693822 wait complete. 2024-03-31T15:56:12.501 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:56:12.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:12.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.644+0000 7f361fa00640 1 Processor -- start 2024-03-31T15:56:12.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.644+0000 7f361fa00640 1 -- start start 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361fa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f361815c120 0x7f361815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361fa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361fa00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36180d2b10 con 0x7f36180d2680 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361fa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3618160bf0 con 0x7f361815c120 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361fa00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3618160350 con 0x7f36180cca30 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f3617e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f36180d2680 0x7f36180b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:43208/0 (socket says 172.21.15.90:43208) 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f3617e00640 1 -- 172.21.15.90:0/918760325 learned_addr learned my addr 172.21.15.90:0/918760325 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361d000640 1 --2- 172.21.15.90:0/918760325 >> v2:172.21.15.102:3300/0 conn(0x7f361815c120 0x7f361815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361da00640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f3617400640 1 -- 172.21.15.90:0/918760325 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127968481 0 0) 0x7f36180d2b10 con 0x7f36180d2680 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f3617400640 1 -- 172.21.15.90:0/918760325 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3600003530 con 0x7f36180d2680 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361d000640 1 -- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 msgr2=0x7f36180ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:12.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361d000640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361d000640 1 -- 172.21.15.90:0/918760325 >> v1:172.21.15.90:6789/0 conn(0x7f36180d2680 legacy=0x7f36180b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361d000640 1 -- 172.21.15.90:0/918760325 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618160630 con 0x7f361815c120 2024-03-31T15:56:12.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.648+0000 7f361da00640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:12.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361d000640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f361815c120 0x7f361815fd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f360c0097c0 tx=0x7f360c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=315c332672d4328f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f3617400640 1 -- 172.21.15.90:0/918760325 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0b0070 con 0x7f361815c120 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f3617400640 1 -- 172.21.15.90:0/918760325 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360c0ab600 con 0x7f361815c120 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f3617400640 1 -- 172.21.15.90:0/918760325 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0ab9b0 con 0x7f361815c120 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 -- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f361815c120 msgr2=0x7f361815fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f361815c120 0x7f361815fd00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f360c0097c0 tx=0x7f360c01c410 comp rx=0 tx=0).stop 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 -- 172.21.15.90:0/918760325 shutdown_connections 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 --2- 172.21.15.90:0/918760325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f361815c120 0x7f361815fd00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 -- 172.21.15.90:0/918760325 >> 172.21.15.90:0/918760325 conn(0x7f361809aa40 msgr2=0x7f361809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 -- 172.21.15.90:0/918760325 shutdown_connections 2024-03-31T15:56:12.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.652+0000 7f361fa00640 1 -- 172.21.15.90:0/918760325 wait complete. 2024-03-31T15:56:12.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 Processor -- start 2024-03-31T15:56:12.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- start start 2024-03-31T15:56:12.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36180d2680 0x7f3618166470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f36180b2f70 con 0x7f361815c120 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36180b2df0 con 0x7f36180d2680 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3618169040 con 0x7f36180cca30 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53124/0 (socket says 172.21.15.90:53124) 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 -- 172.21.15.90:0/1842554952 learned_addr learned my addr 172.21.15.90:0/1842554952 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f3617e00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36180d2680 0x7f3618166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361d000640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 -- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 msgr2=0x7f36180aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 -- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36180d2680 msgr2=0x7f3618166470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36180d2680 0x7f3618166470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 -- 172.21.15.90:0/1842554952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618153c20 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361da00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3610009e60 tx=0x7f361000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610018030 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f361000cdf0 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3618153f00 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610003bc0 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361806e370 con 0x7f361815c120 2024-03-31T15:56:12.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.656+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361815f810 con 0x7f361815c120 2024-03-31T15:56:12.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.660+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f361001b070 con 0x7f361815c120 2024-03-31T15:56:12.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.660+0000 7f3614c00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 0x7f35f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:12.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.660+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7f3610087680 con 0x7f361815c120 2024-03-31T15:56:12.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.660+0000 7f361d000640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 0x7f35f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:12.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.660+0000 7f361d000640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 0x7f35f806a840 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f360c0ad000 tx=0x7f360c0a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:12.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.664+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3610010040 con 0x7f361815c120 2024-03-31T15:56:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36180b1b60 con 0x7f361815c120 2024-03-31T15:56:12.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f3614c00640 1 -- 172.21.15.90:0/1842554952 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5186 (secure 0 0 0) 0x7f3610057900 con 0x7f361815c120 2024-03-31T15:56:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:12.854 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:31.990711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:56:12.854 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 msgr2=0x7f35f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 0x7f35f806a840 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f360c0ad000 tx=0x7f360c0a4f90 comp rx=0 tx=0).stop 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 msgr2=0x7f361806c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3610009e60 tx=0x7f361000c490 comp rx=0 tx=0).stop 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 shutdown_connections 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f35f8068380 0x7f35f806a840 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f361815c120 0x7f361806c700 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36180d2680 0x7f3618166470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 --2- 172.21.15.90:0/1842554952 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f36180cca30 0x7f36180aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 >> 172.21.15.90:0/1842554952 conn(0x7f361809aa40 msgr2=0x7f361806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.860+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 shutdown_connections 2024-03-31T15:56:12.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:12.864+0000 7f361fa00640 1 -- 172.21.15.90:0/1842554952 wait complete. 2024-03-31T15:56:12.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.016+0000 7ff101000640 1 Processor -- start 2024-03-31T15:56:13.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.016+0000 7ff101000640 1 -- start start 2024-03-31T15:56:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff101000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0fc073590 0x7ff0fc15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff101000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff0fc15b830 0x7ff0fc0b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff101000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc06ce30 con 0x7ff0fc15af10 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff101000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc06eff0 con 0x7ff0fc15b830 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff101000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc06f2c0 con 0x7ff0fc073590 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fbe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff0fc15b830 0x7ff0fc0b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fb400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0fc15af10 0x7ff0fc06e8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39496/0 (socket says 172.21.15.90:39496) 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fb400640 1 -- 172.21.15.90:0/2513376249 learned_addr learned my addr 172.21.15.90:0/2513376249 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:13.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0faa00640 1 --2- 172.21.15.90:0/2513376249 >> v2:172.21.15.102:3300/0 conn(0x7ff0fc073590 0x7ff0fc15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fbe00640 1 -- 172.21.15.90:0/2513376249 >> v2:172.21.15.102:3300/0 conn(0x7ff0fc073590 msgr2=0x7ff0fc15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fbe00640 1 --2- 172.21.15.90:0/2513376249 >> v2:172.21.15.102:3300/0 conn(0x7ff0fc073590 0x7ff0fc15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fbe00640 1 -- 172.21.15.90:0/2513376249 >> v1:172.21.15.90:6789/0 conn(0x7ff0fc15af10 legacy=0x7ff0fc06e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fbe00640 1 -- 172.21.15.90:0/2513376249 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc06f5a0 con 0x7ff0fc15b830 2024-03-31T15:56:13.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.020+0000 7ff0fa000640 1 -- 172.21.15.90:0/2513376249 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138985819 0 0) 0x7ff0fc06ce30 con 0x7ff0fc15af10 2024-03-31T15:56:13.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fbe00640 1 --2- 172.21.15.90:0/2513376249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc15b830 0x7ff0fc0b31e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0cd2c0 tx=0x7ff0f401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5f643579dc1ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fa000640 1 -- 172.21.15.90:0/2513376249 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40b2070 con 0x7ff0fc15b830 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fa000640 1 -- 172.21.15.90:0/2513376249 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f40a5930 con 0x7ff0fc15b830 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fa000640 1 -- 172.21.15.90:0/2513376249 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40a5d00 con 0x7ff0fc15b830 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- 172.21.15.90:0/2513376249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc15b830 msgr2=0x7ff0fc0b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- 172.21.15.90:0/2513376249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc15b830 0x7ff0fc0b31e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0cd2c0 tx=0x7ff0f401c870 comp rx=0 tx=0).stop 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- 172.21.15.90:0/2513376249 shutdown_connections 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- 172.21.15.90:0/2513376249 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc15b830 0x7ff0fc0b31e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- 172.21.15.90:0/2513376249 >> v2:172.21.15.102:3300/0 conn(0x7ff0fc073590 0x7ff0fc15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- 172.21.15.90:0/2513376249 >> 172.21.15.90:0/2513376249 conn(0x7ff0fc0cff30 msgr2=0x7ff0fc0d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- 172.21.15.90:0/2513376249 shutdown_connections 2024-03-31T15:56:13.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- 172.21.15.90:0/2513376249 wait complete. 2024-03-31T15:56:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 Processor -- start 2024-03-31T15:56:13.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- start start 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0fc15af10 0x7ff0fc154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0fc15b830 0x7ff0fc160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc15be40 con 0x7ff0fc153170 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc15bcc0 con 0x7ff0fc15b830 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff101000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc0b3f30 con 0x7ff0fc15af10 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48206/0 (socket says 172.21.15.90:48206) 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.024+0000 7ff0fbe00640 1 -- 172.21.15.90:0/4105512885 learned_addr learned my addr 172.21.15.90:0/4105512885 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fb400640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0fc15af10 0x7ff0fc154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0faa00640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0fc15b830 0x7ff0fc160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 -- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0fc15af10 msgr2=0x7ff0fc154b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0fc15af10 0x7ff0fc154b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 -- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0fc15b830 msgr2=0x7ff0fc160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0fc15b830 0x7ff0fc160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.021 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 -- 172.21.15.90:0/4105512885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc15ea90 con 0x7ff0fc153170 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0fbe00640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff0f401a4a0 tx=0x7ff0f4019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40b2050 con 0x7ff0fc153170 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f40be070 con 0x7ff0fc153170 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40ada90 con 0x7ff0fc153170 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc15ed70 con 0x7ff0fc153170 2024-03-31T15:56:13.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc15e720 con 0x7ff0fc153170 2024-03-31T15:56:13.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc068d00 con 0x7ff0fc153170 2024-03-31T15:56:13.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0f40adc30 con 0x7ff0fc153170 2024-03-31T15:56:13.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 0x7ff0dc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.028+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7ff0f4121680 con 0x7ff0fc153170 2024-03-31T15:56:13.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.032+0000 7ff0fb400640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 0x7ff0dc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.032+0000 7ff0fb400640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 0x7ff0dc06a870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec007fd0 tx=0x7ff0ec007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.032+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff0f40c7020 con 0x7ff0fc153170 2024-03-31T15:56:13.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0fc0d0790 con 0x7ff0fc153170 2024-03-31T15:56:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff0f3400640 1 -- 172.21.15.90:0/4105512885 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5186 (secure 0 0 0) 0x7ff0f4016520 con 0x7ff0fc153170 2024-03-31T15:56:13.217 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:13.217 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:55:17.930644+0000","modified":"2024-03-31T15:55:31.990711+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5502},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5502":{"gid":5502,"name":"c","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6834/353199836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":353199836},{"type":"v1","addr":"172.21.15.90:6834","nonce":353199836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5502,"qdb_cluster":[5502]},"id":2}]} 2024-03-31T15:56:13.217 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 18 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 msgr2=0x7ff0dc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 0x7ff0dc06a870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec007fd0 tx=0x7ff0ec007d60 comp rx=0 tx=0).stop 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 msgr2=0x7ff0fc161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff0f401a4a0 tx=0x7ff0f4019610 comp rx=0 tx=0).stop 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 shutdown_connections 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff0dc0683b0 0x7ff0dc06a870 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff0fc153170 0x7ff0fc161110 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0fc15b830 0x7ff0fc160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 --2- 172.21.15.90:0/4105512885 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff0fc15af10 0x7ff0fc154b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 >> 172.21.15.90:0/4105512885 conn(0x7ff0fc0cff30 msgr2=0x7ff0fc06d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 shutdown_connections 2024-03-31T15:56:13.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.224+0000 7ff101000640 1 -- 172.21.15.90:0/4105512885 wait complete. 2024-03-31T15:56:13.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:13.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.380+0000 7fac19e00640 1 Processor -- start 2024-03-31T15:56:13.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.380+0000 7fac19e00640 1 -- start start 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac19e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac1406d7c0 0x7fac1415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac19e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1415f780 con 0x7fac1415e1b0 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac19e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1415f1c0 con 0x7fac1406c7b0 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac19e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fac141622b0 con 0x7fac1406d7c0 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac1406d7c0 0x7fac1415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fac1406d7c0 0x7fac1415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48210/0 (socket says 172.21.15.90:48210) 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac12a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fac1415e1b0 0x7fac14162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39500/0 (socket says 172.21.15.90:39500) 2024-03-31T15:56:13.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 -- 172.21.15.90:0/2798998155 learned_addr learned my addr 172.21.15.90:0/2798998155 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:13.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 -- 172.21.15.90:0/2798998155 >> v1:172.21.15.102:6789/0 conn(0x7fac1406c7b0 legacy=0x7fac1406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 -- 172.21.15.90:0/2798998155 >> v1:172.21.15.90:6789/0 conn(0x7fac1415e1b0 legacy=0x7fac14162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac12000640 1 -- 172.21.15.90:0/2798998155 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292998886 0 0) 0x7fac1415f1c0 con 0x7fac1406c7b0 2024-03-31T15:56:13.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.384+0000 7fac13400640 1 -- 172.21.15.90:0/2798998155 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1415f4a0 con 0x7fac1406d7c0 2024-03-31T15:56:13.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac13400640 1 --2- 172.21.15.90:0/2798998155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406d7c0 0x7fac1415e590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc272fdc70e7b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac12000640 1 -- 172.21.15.90:0/2798998155 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0a5640 con 0x7fac1406d7c0 2024-03-31T15:56:13.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac12000640 1 -- 172.21.15.90:0/2798998155 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c0a57e0 con 0x7fac1406d7c0 2024-03-31T15:56:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac12000640 1 -- 172.21.15.90:0/2798998155 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0a5bb0 con 0x7fac1406d7c0 2024-03-31T15:56:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- 172.21.15.90:0/2798998155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406d7c0 msgr2=0x7fac1415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 --2- 172.21.15.90:0/2798998155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406d7c0 0x7fac1415e590 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c01c800 comp rx=0 tx=0).stop 2024-03-31T15:56:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- 172.21.15.90:0/2798998155 shutdown_connections 2024-03-31T15:56:13.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 --2- 172.21.15.90:0/2798998155 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406d7c0 0x7fac1415e590 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- 172.21.15.90:0/2798998155 >> 172.21.15.90:0/2798998155 conn(0x7fac1409aa40 msgr2=0x7fac1409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- 172.21.15.90:0/2798998155 shutdown_connections 2024-03-31T15:56:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- 172.21.15.90:0/2798998155 wait complete. 2024-03-31T15:56:13.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 Processor -- start 2024-03-31T15:56:13.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- start start 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac1406d7c0 0x7fac140b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac1415e1b0 0x7fac140b2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fac140ce7a0 con 0x7fac1406c7b0 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac140d2a90 con 0x7fac1415e1b0 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.388+0000 7fac19e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fac140ce920 con 0x7fac1406d7c0 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48214/0 (socket says 172.21.15.90:48214) 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 -- 172.21.15.90:0/3050328755 learned_addr learned my addr 172.21.15.90:0/3050328755 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:13.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac0a000640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac1406d7c0 0x7fac140b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac13400640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac1415e1b0 0x7fac140b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 -- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac1406d7c0 msgr2=0x7fac140b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac1406d7c0 0x7fac140b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 -- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac1415e1b0 msgr2=0x7fac140b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac1415e1b0 0x7fac140b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 -- 172.21.15.90:0/3050328755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac140b01f0 con 0x7fac1406c7b0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac12a00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fac0400c780 tx=0x7fac0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04018070 con 0x7fac1406c7b0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac040072d0 con 0x7fac1406c7b0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac040047f0 con 0x7fac1406c7b0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac14153620 con 0x7fac1406c7b0 2024-03-31T15:56:13.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac140b0770 con 0x7fac1406c7b0 2024-03-31T15:56:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac140686d0 con 0x7fac1406c7b0 2024-03-31T15:56:13.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.392+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fac0401b070 con 0x7fac1406c7b0 2024-03-31T15:56:13.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.396+0000 7fac0be00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 0x7fabf006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.396+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fac040860c0 con 0x7fac1406c7b0 2024-03-31T15:56:13.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.396+0000 7fac0a000640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 0x7fabf006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.396+0000 7fac0a000640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 0x7fabf006a870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fac140ced50 tx=0x7fabfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.400+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fac04055d20 con 0x7fac1406c7b0 2024-03-31T15:56:13.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.572+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac1416afa0 con 0x7fac1406c7b0 2024-03-31T15:56:13.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.572+0000 7fac0be00640 1 -- 172.21.15.90:0/3050328755 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18767 (secure 0 0 0) 0x7fac0405c020 con 0x7fac1406c7b0 2024-03-31T15:56:13.566 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:13.566 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":43,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:55:33.624447+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:55:15.583234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-31T15:55:16.891180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-31T15:55:31.097739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1c","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 msgr2=0x7fabf006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 0x7fabf006a870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fac140ced50 tx=0x7fabfc009040 comp rx=0 tx=0).stop 2024-03-31T15:56:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 msgr2=0x7fac141530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fac0400c780 tx=0x7fac0400cc40 comp rx=0 tx=0).stop 2024-03-31T15:56:13.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 shutdown_connections 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fabf00683b0 0x7fabf006a870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac1415e1b0 0x7fac140b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fac1406d7c0 0x7fac140b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 --2- 172.21.15.90:0/3050328755 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fac1406c7b0 0x7fac141530e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 >> 172.21.15.90:0/3050328755 conn(0x7fac1409aa40 msgr2=0x7fac1415fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 shutdown_connections 2024-03-31T15:56:13.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.576+0000 7fac19e00640 1 -- 172.21.15.90:0/3050328755 wait complete. 2024-03-31T15:56:13.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:56:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 Processor -- start 2024-03-31T15:56:13.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 -- start start 2024-03-31T15:56:13.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6c00d0d20 0x7fe6c00d2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c00d1190 con 0x7fe6c01624a0 2024-03-31T15:56:13.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c00d1400 con 0x7fe6c015e320 2024-03-31T15:56:13.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6c6200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe6c015e700 con 0x7fe6c00d0d20 2024-03-31T15:56:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6c00d0d20 0x7fe6c00d2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe6c00d0d20 0x7fe6c00d2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48226/0 (socket says 172.21.15.90:48226) 2024-03-31T15:56:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 -- 172.21.15.90:0/4132866540 learned_addr learned my addr 172.21.15.90:0/4132866540 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 -- 172.21.15.90:0/4132866540 >> v1:172.21.15.102:6789/0 conn(0x7fe6c015e320 legacy=0x7fe6c0161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 -- 172.21.15.90:0/4132866540 >> v1:172.21.15.90:6789/0 conn(0x7fe6c01624a0 legacy=0x7fe6c00b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.732+0000 7fe6bfe00640 1 -- 172.21.15.90:0/4132866540 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c00b1a20 con 0x7fe6c00d0d20 2024-03-31T15:56:13.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6bfe00640 1 --2- 172.21.15.90:0/4132866540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c00d2940 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6cb9a65871a55b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6be000640 1 -- 172.21.15.90:0/4132866540 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80a5920 con 0x7fe6c00d0d20 2024-03-31T15:56:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6be000640 1 -- 172.21.15.90:0/4132866540 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b80a5ac0 con 0x7fe6c00d0d20 2024-03-31T15:56:13.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6be000640 1 -- 172.21.15.90:0/4132866540 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80ad3f0 con 0x7fe6c00d0d20 2024-03-31T15:56:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 -- 172.21.15.90:0/4132866540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 msgr2=0x7fe6c00d2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 --2- 172.21.15.90:0/4132866540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c00d2940 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009e50 tx=0x7fe6b801ca30 comp rx=0 tx=0).stop 2024-03-31T15:56:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 -- 172.21.15.90:0/4132866540 shutdown_connections 2024-03-31T15:56:13.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 --2- 172.21.15.90:0/4132866540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c00d2940 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 -- 172.21.15.90:0/4132866540 >> 172.21.15.90:0/4132866540 conn(0x7fe6c008bfd0 msgr2=0x7fe6c00d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 -- 172.21.15.90:0/4132866540 shutdown_connections 2024-03-31T15:56:13.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.736+0000 7fe6c6200640 1 -- 172.21.15.90:0/4132866540 wait complete. 2024-03-31T15:56:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 Processor -- start 2024-03-31T15:56:13.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- start start 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 0x7fe6c00cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe6c01624a0 0x7fe6c00cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c00b37b0 con 0x7fe6c00d0d20 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c00b3630 con 0x7fe6c015e320 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c016bf10 con 0x7fe6c01624a0 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48238/0 (socket says 172.21.15.90:48238) 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 -- 172.21.15.90:0/1123887489 learned_addr learned my addr 172.21.15.90:0/1123887489 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:13.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bea00640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 0x7fe6c00cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bfe00640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe6c01624a0 0x7fe6c00cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 -- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe6c01624a0 msgr2=0x7fe6c00cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe6c01624a0 0x7fe6c00cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 -- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 msgr2=0x7fe6c00cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 0x7fe6c00cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 -- 172.21.15.90:0/1123887489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c00c9070 con 0x7fe6c00d0d20 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bea00640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 0x7fe6c00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6bf400640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c3f0 tx=0x7fe6b000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0017070 con 0x7fe6c00d0d20 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b00052d0 con 0x7fe6c00d0d20 2024-03-31T15:56:13.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c00c9350 con 0x7fe6c00d0d20 2024-03-31T15:56:13.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0003840 con 0x7fe6c00d0d20 2024-03-31T15:56:13.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c00c9910 con 0x7fe6c00d0d20 2024-03-31T15:56:13.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.740+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6c0068660 con 0x7fe6c00d0d20 2024-03-31T15:56:13.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.744+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe6b001a070 con 0x7fe6c00d0d20 2024-03-31T15:56:13.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.744+0000 7fe6b7400640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 0x7fe6a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:13.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.744+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6525+0+0 (secure 0 0 0) 0x7fe6b0086080 con 0x7fe6c00d0d20 2024-03-31T15:56:13.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.744+0000 7fe6bea00640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 0x7fe6a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:13.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.744+0000 7fe6bea00640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 0x7fe6a006a870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe6c01fa730 tx=0x7fe6ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:13.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.748+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe6b0005480 con 0x7fe6c00d0d20 2024-03-31T15:56:13.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.940+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe6c006d360 con 0x7fe6c00d0d20 2024-03-31T15:56:13.935 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-31T15:56:13.940+0000 7fc737200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:56:13.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.944+0000 7fe6b7400640 1 -- 172.21.15.90:0/1123887489 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7fe6b0056310 con 0x7fe6c00d0d20 2024-03-31T15:56:13.938 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 msgr2=0x7fe6a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 0x7fe6a006a870 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe6c01fa730 tx=0x7fe6ac009040 comp rx=0 tx=0).stop 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 msgr2=0x7fe6c01501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c3f0 tx=0x7fe6b000c8b0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 shutdown_connections 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe6a00683b0 0x7fe6a006a870 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe6c01624a0 0x7fe6c00cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6c015e320 0x7fe6c00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 --2- 172.21.15.90:0/1123887489 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe6c00d0d20 0x7fe6c01501c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 >> 172.21.15.90:0/1123887489 conn(0x7fe6c008bfd0 msgr2=0x7fe6c0070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 shutdown_connections 2024-03-31T15:56:13.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:13.948+0000 7fe6c6200640 1 -- 172.21.15.90:0/1123887489 wait complete. 2024-03-31T15:56:13.949 INFO:tasks.ceph.mds.c.smithi090.stderr: -4307> 2024-03-31T15:55:21.512+0000 7f7908c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:56:13.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:56:13.997 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-31T15:56:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 Processor -- start 2024-03-31T15:56:14.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 -- start start 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f944c0725a0 0x7f944c15b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f944c0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f944c072a60 con 0x7f944c15b040 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f944c0ae890 con 0x7f944c0725a0 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9453400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f944c072d40 con 0x7f944c071940 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9450a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f944c15b040 0x7f944c071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39528/0 (socket says 172.21.15.90:39528) 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9450a00640 1 -- 172.21.15.90:0/2630480652 learned_addr learned my addr 172.21.15.90:0/2630480652 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 --2- 172.21.15.90:0/2630480652 >> v2:172.21.15.102:3300/0 conn(0x7f944c0725a0 0x7f944c15b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9451400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f944c0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348700687 0 0) 0x7f944c072a60 con 0x7f944c15b040 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9430003530 con 0x7f944c15b040 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2121269172 0 0) 0x7f9430003530 con 0x7f944c15b040 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 msgr2=0x7f944c0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f944c0b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 >> v2:172.21.15.102:3300/0 conn(0x7f944c0725a0 msgr2=0x7f944c15b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 --2- 172.21.15.90:0/2630480652 >> v2:172.21.15.102:3300/0 conn(0x7f944c0725a0 0x7f944c15b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944c0aeac0 con 0x7f944c15b040 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f943c001a00 con 0x7f944c15b040 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 >> v1:172.21.15.90:6789/0 conn(0x7f944c15b040 legacy=0x7f944c071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9430004810 0x7f9430004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f94300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f944c072d40 con 0x7f9430004810 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f943000b9d0 con 0x7f94300039c0 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f943000bb50 con 0x7f944c071940 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9430004810 0x7f9430004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9450a00640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f9451400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f94300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 -- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 msgr2=0x7f94300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f944c071940 0x7f94300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 -- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94300039c0 msgr2=0x7f9430003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.108+0000 7f944be00640 1 -- 172.21.15.90:0/2630480652 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9430003530 con 0x7f9430004810 2024-03-31T15:56:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f944be00640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9430004810 0x7f9430004c90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9438009ae0 tx=0x7f943801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb49e22af21a2ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:14.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94380b2070 con 0x7f9430004810 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94380a55c0 con 0x7f9430004810 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f944b400640 1 -- 172.21.15.90:0/2630480652 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94380096b0 con 0x7f9430004810 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 -- 172.21.15.90:0/2630480652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9430004810 msgr2=0x7f9430004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 --2- 172.21.15.90:0/2630480652 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9430004810 0x7f9430004c90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9438009ae0 tx=0x7f943801cab0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9451400640 1 -- 172.21.15.90:0/2630480652 reap_dead start 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 -- 172.21.15.90:0/2630480652 shutdown_connections 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 -- 172.21.15.90:0/2630480652 >> 172.21.15.90:0/2630480652 conn(0x7f944c0b18b0 msgr2=0x7f944c0b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 -- 172.21.15.90:0/2630480652 shutdown_connections 2024-03-31T15:56:14.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.112+0000 7f9453400640 1 -- 172.21.15.90:0/2630480652 wait complete. 2024-03-31T15:56:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 Processor -- start 2024-03-31T15:56:14.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- start start 2024-03-31T15:56:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9430004810 0x7f944c0b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944c1f4780 0x7f944c1f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f944c06f020 con 0x7f944c14f5f0 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f944c0b35c0 con 0x7f944c1f4780 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f944c0713f0 con 0x7f9430004810 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48270/0 (socket says 172.21.15.90:48270) 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 -- 172.21.15.90:0/152579646 learned_addr learned my addr 172.21.15.90:0/152579646 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f944be00640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944c1f4780 0x7f944c1f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9450a00640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9430004810 0x7f944c0b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 -- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9430004810 msgr2=0x7f944c0b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9430004810 0x7f944c0b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 -- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944c1f4780 msgr2=0x7f944c1f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944c1f4780 0x7f944c1f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 -- 172.21.15.90:0/152579646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944c1f1fb0 con 0x7f944c14f5f0 2024-03-31T15:56:14.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9451400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f944c070e00 tx=0x7f9444011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444004240 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944401b070 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944c1f2250 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944400f770 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.116+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944c1f2810 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944c0685f0 con 0x7f944c14f5f0 2024-03-31T15:56:14.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9444012040 con 0x7f944c14f5f0 2024-03-31T15:56:14.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9448c00640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 0x7f942c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:14.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6611+0+0 (secure 0 0 0) 0x7f9444086ab0 con 0x7f944c14f5f0 2024-03-31T15:56:14.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9450a00640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 0x7f942c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:14.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.120+0000 7f9450a00640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 0x7f942c06a870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f943c006c50 tx=0x7f943c00a060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:14.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.124+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9444056c40 con 0x7f944c14f5f0 2024-03-31T15:56:14.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.316+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f944c0b2320 con 0x7f944c14f5f0 2024-03-31T15:56:14.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.944+0000 7f9448c00640 1 -- 172.21.15.90:0/152579646 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v20) ==== 104+0+0 (secure 0 0 0) 0x7f9444056540 con 0x7f944c14f5f0 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 msgr2=0x7f942c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 0x7f942c06a870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f943c006c50 tx=0x7f943c00a060 comp rx=0 tx=0).stop 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 msgr2=0x7f944c14fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f944c070e00 tx=0x7f9444011cf0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 shutdown_connections 2024-03-31T15:56:14.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f942c0683b0 0x7f942c06a870 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f944c14f5f0 0x7f944c14fa70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f944c1f4780 0x7f944c1f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 --2- 172.21.15.90:0/152579646 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9430004810 0x7f944c0b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 >> 172.21.15.90:0/152579646 conn(0x7f944c0b18b0 msgr2=0x7f944c0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 shutdown_connections 2024-03-31T15:56:14.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:14.948+0000 7f9453400640 1 -- 172.21.15.90:0/152579646 wait complete. 2024-03-31T15:56:14.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:56:15.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 Processor -- start 2024-03-31T15:56:15.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 -- start start 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 0x7f676406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67640cbab0 0x7f676415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f676415cd60 con 0x7f67640cb630 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f676415fdc0 con 0x7f67640cbab0 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f676b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f676415d040 con 0x7f676415be10 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67640cbab0 0x7f676415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67640cbab0 0x7f676415c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48282/0 (socket says 172.21.15.90:48282) 2024-03-31T15:56:15.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 -- 172.21.15.90:0/2442118116 learned_addr learned my addr 172.21.15.90:0/2442118116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 -- 172.21.15.90:0/2442118116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 msgr2=0x7f676406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6763e00640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 0x7f676406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 0x7f676406ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 -- 172.21.15.90:0/2442118116 >> v1:172.21.15.102:6789/0 conn(0x7f67640cb630 legacy=0x7f6764160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6769200640 1 -- 172.21.15.90:0/2442118116 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676415d320 con 0x7f67640cbab0 2024-03-31T15:56:15.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.152+0000 7f6763e00640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 0x7f676406ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:15.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f6769200640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67640cbab0 0x7f676415c1f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f675c009870 tx=0x7f675c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3928250010bf439 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:15.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f6762a00640 1 -- 172.21.15.90:0/2442118116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c0b0070 con 0x7f67640cbab0 2024-03-31T15:56:15.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f6762a00640 1 -- 172.21.15.90:0/2442118116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675c0ab600 con 0x7f67640cbab0 2024-03-31T15:56:15.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f6762a00640 1 -- 172.21.15.90:0/2442118116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c0ab9d0 con 0x7f67640cbab0 2024-03-31T15:56:15.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 -- 172.21.15.90:0/2442118116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67640cbab0 msgr2=0x7f676415c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:15.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67640cbab0 0x7f676415c1f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f675c009870 tx=0x7f675c01c490 comp rx=0 tx=0).stop 2024-03-31T15:56:15.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 -- 172.21.15.90:0/2442118116 shutdown_connections 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67640cbab0 0x7f676415c1f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 --2- 172.21.15.90:0/2442118116 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f676415be10 0x7f676406ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 -- 172.21.15.90:0/2442118116 >> 172.21.15.90:0/2442118116 conn(0x7f67640b1880 msgr2=0x7f67640b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 -- 172.21.15.90:0/2442118116 shutdown_connections 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.156+0000 7f676b200640 1 -- 172.21.15.90:0/2442118116 wait complete. 2024-03-31T15:56:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 Processor -- start 2024-03-31T15:56:15.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- start start 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 0x7f676415e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f67640cbab0 0x7f676415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6764169160 con 0x7f676415be10 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f67641692e0 con 0x7f67640cb630 2024-03-31T15:56:15.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6764169460 con 0x7f67640cbab0 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48292/0 (socket says 172.21.15.90:48292) 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 -- 172.21.15.90:0/3364782576 learned_addr learned my addr 172.21.15.90:0/3364782576 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6763e00640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 0x7f676415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6763400640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f67640cbab0 0x7f676415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 -- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f67640cbab0 msgr2=0x7f676415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f67640cbab0 0x7f676415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 -- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 msgr2=0x7f676415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 0x7f676415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 -- 172.21.15.90:0/3364782576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676415d9b0 con 0x7f676415be10 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6763e00640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 0x7f676415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f6769200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f675c0ae040 tx=0x7f675c019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c0b0040 con 0x7f676415be10 2024-03-31T15:56:15.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675c0ba070 con 0x7f676415be10 2024-03-31T15:56:15.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c0aa840 con 0x7f676415be10 2024-03-31T15:56:15.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67641f7ec0 con 0x7f676415be10 2024-03-31T15:56:15.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.160+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67640c2eb0 con 0x7f676415be10 2024-03-31T15:56:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67640686d0 con 0x7f676415be10 2024-03-31T15:56:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f675c0aa9e0 con 0x7f676415be10 2024-03-31T15:56:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f675be00640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 0x7f674406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6611+0+0 (secure 0 0 0) 0x7f675c120c80 con 0x7f676415be10 2024-03-31T15:56:15.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f6763e00640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 0x7f674406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:15.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.164+0000 7f6763e00640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 0x7f674406a840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f675400e3d0 tx=0x7f6754007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:15.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.168+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f675c016110 con 0x7f676415be10 2024-03-31T15:56:15.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.328+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f676406e6a0 con 0x7f676415be10 2024-03-31T15:56:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.952+0000 7f675be00640 1 -- 172.21.15.90:0/3364782576 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) ==== 181+0+0 (secure 0 0 0) 0x7f675c0f0f40 con 0x7f676415be10 2024-03-31T15:56:15.945 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:56:15.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 msgr2=0x7f674406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 0x7f674406a840 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f675400e3d0 tx=0x7f6754007a10 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 msgr2=0x7f67641f89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f675c0ae040 tx=0x7f675c019230 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 shutdown_connections 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6744068380 0x7f674406a840 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676415be10 0x7f67641f89a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f67640cbab0 0x7f676415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 --2- 172.21.15.90:0/3364782576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67640cb630 0x7f676415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 >> 172.21.15.90:0/3364782576 conn(0x7f67640b1880 msgr2=0x7f67641555a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 shutdown_connections 2024-03-31T15:56:15.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:15.956+0000 7f676b200640 1 -- 172.21.15.90:0/3364782576 wait complete. 2024-03-31T15:56:15.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:56:16.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 Processor -- start 2024-03-31T15:56:16.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 -- start start 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 0x7fe2900b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe29015dff0 0x7fe29015e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2900d1b90 con 0x7fe29006b850 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe290161e30 con 0x7fe29015dff0 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe296000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe290162110 con 0x7fe29006bc30 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28f400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe29006b850 0x7fe2900b3dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39540/0 (socket says 172.21.15.90:39540) 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28f400640 1 -- 172.21.15.90:0/1684525845 learned_addr learned my addr 172.21.15.90:0/1684525845 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:16.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28fe00640 1 --2- 172.21.15.90:0/1684525845 >> v2:172.21.15.90:3300/0 conn(0x7fe29015dff0 0x7fe29015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28e000640 1 -- 172.21.15.90:0/1684525845 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416571517 0 0) 0x7fe2900d1b90 con 0x7fe29006b850 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28ea00640 1 --2- 172.21.15.90:0/1684525845 >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 0x7fe2900b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28e000640 1 -- 172.21.15.90:0/1684525845 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe274003530 con 0x7fe29006b850 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28fe00640 1 -- 172.21.15.90:0/1684525845 >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 msgr2=0x7fe2900b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28fe00640 1 --2- 172.21.15.90:0/1684525845 >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 0x7fe2900b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28fe00640 1 -- 172.21.15.90:0/1684525845 >> v1:172.21.15.90:6789/0 conn(0x7fe29006b850 legacy=0x7fe2900b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28fe00640 1 -- 172.21.15.90:0/1684525845 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2901623f0 con 0x7fe29015dff0 2024-03-31T15:56:16.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.116+0000 7fe28ea00640 1 --2- 172.21.15.90:0/1684525845 >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 0x7fe2900b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe28fe00640 1 --2- 172.21.15.90:0/1684525845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29015dff0 0x7fe29015e400 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe2900722c0 tx=0x7fe28801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8fe90d05ddc2f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe28e000640 1 -- 172.21.15.90:0/1684525845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2880b2070 con 0x7fe29015dff0 2024-03-31T15:56:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe28e000640 1 -- 172.21.15.90:0/1684525845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2880a5870 con 0x7fe29015dff0 2024-03-31T15:56:16.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe28e000640 1 -- 172.21.15.90:0/1684525845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2880a5c40 con 0x7fe29015dff0 2024-03-31T15:56:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 -- 172.21.15.90:0/1684525845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29015dff0 msgr2=0x7fe29015e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 --2- 172.21.15.90:0/1684525845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29015dff0 0x7fe29015e400 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe2900722c0 tx=0x7fe28801c930 comp rx=0 tx=0).stop 2024-03-31T15:56:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 -- 172.21.15.90:0/1684525845 shutdown_connections 2024-03-31T15:56:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 --2- 172.21.15.90:0/1684525845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29015dff0 0x7fe29015e400 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 --2- 172.21.15.90:0/1684525845 >> v2:172.21.15.102:3300/0 conn(0x7fe29006bc30 0x7fe2900b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 -- 172.21.15.90:0/1684525845 >> 172.21.15.90:0/1684525845 conn(0x7fe29006c700 msgr2=0x7fe29006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 -- 172.21.15.90:0/1684525845 shutdown_connections 2024-03-31T15:56:16.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.120+0000 7fe296000640 1 -- 172.21.15.90:0/1684525845 wait complete. 2024-03-31T15:56:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 Processor -- start 2024-03-31T15:56:16.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- start start 2024-03-31T15:56:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe29006bc30 0x7fe290159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe29015dff0 0x7fe2901503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe29015f2d0 con 0x7fe29006b850 2024-03-31T15:56:16.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe29015f150 con 0x7fe29015dff0 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe29015f450 con 0x7fe29006bc30 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48304/0 (socket says 172.21.15.90:48304) 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 -- 172.21.15.90:0/1123146339 learned_addr learned my addr 172.21.15.90:0/1123146339 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 -- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe29006bc30 msgr2=0x7fe290159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe29006bc30 0x7fe290159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 -- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe29015dff0 msgr2=0x7fe2901503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe29015dff0 0x7fe2901503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 -- 172.21.15.90:0/1123146339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe290150900 con 0x7fe29006b850 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe28f400640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe28000c450 tx=0x7fe28000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280017070 con 0x7fe29006b850 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe280005cf0 con 0x7fe29006b850 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280007290 con 0x7fe29006b850 2024-03-31T15:56:16.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2900ce3f0 con 0x7fe29006b850 2024-03-31T15:56:16.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2900ce9b0 con 0x7fe29006b850 2024-03-31T15:56:16.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.124+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2900686d0 con 0x7fe29006b850 2024-03-31T15:56:16.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.128+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe28001a070 con 0x7fe29006b850 2024-03-31T15:56:16.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.128+0000 7fe287400640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 0x7fe27006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:16.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.128+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6173+0+0 (secure 0 0 0) 0x7fe280085fd0 con 0x7fe29006b850 2024-03-31T15:56:16.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.128+0000 7fe28ea00640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 0x7fe27006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:16.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.128+0000 7fe28ea00640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 0x7fe27006a840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe2900d03c0 tx=0x7fe27c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:16.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.132+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe280056320 con 0x7fe29006b850 2024-03-31T15:56:16.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.288+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe290072b80 con 0x7fe29006b850 2024-03-31T15:56:16.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.956+0000 7fe287400640 1 -- 172.21.15.90:0/1123146339 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) ==== 169+0+0 (secure 0 0 0) 0x7fe280055c20 con 0x7fe29006b850 2024-03-31T15:56:16.952 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-31T15:56:16.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 msgr2=0x7fe27006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 0x7fe27006a840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe2900d03c0 tx=0x7fe27c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:16.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 msgr2=0x7fe2900cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:16.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe28000c450 tx=0x7fe28000c910 comp rx=0 tx=0).stop 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 shutdown_connections 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe270068380 0x7fe27006a840 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe29015dff0 0x7fe2901503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe29006bc30 0x7fe290159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 --2- 172.21.15.90:0/1123146339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe29006b850 0x7fe2900cfe80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 >> 172.21.15.90:0/1123146339 conn(0x7fe29006c700 msgr2=0x7fe29006fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 shutdown_connections 2024-03-31T15:56:16.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:16.960+0000 7fe296000640 1 -- 172.21.15.90:0/1123146339 wait complete. 2024-03-31T15:56:16.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-31T15:56:17.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 Processor -- start 2024-03-31T15:56:17.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 -- start start 2024-03-31T15:56:17.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f739c15be10 0x7f739c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c15c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739c15cd60 con 0x7f739c0cb630 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f739c15fdc0 con 0x7f739c15be10 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a4600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f739c15d040 con 0x7f739c0cbae0 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a2600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c15c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f739c15be10 0x7f739c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f739c15be10 0x7f739c06ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:60312/0 (socket says 172.21.15.90:60312) 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 -- 172.21.15.90:0/751358477 learned_addr learned my addr 172.21.15.90:0/751358477 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 -- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 msgr2=0x7f739c15c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 --2- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c15c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 -- 172.21.15.90:0/751358477 >> v1:172.21.15.102:6789/0 conn(0x7f739c0cb630 legacy=0x7f739c160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.124+0000 7f73a1c00640 1 -- 172.21.15.90:0/751358477 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c15d320 con 0x7f739c15be10 2024-03-31T15:56:17.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a1c00640 1 --2- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c06ea10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7390009870 tx=0x7f739001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c7843eac58541c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f739be00640 1 -- 172.21.15.90:0/751358477 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900b1070 con 0x7f739c15be10 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f739be00640 1 -- 172.21.15.90:0/751358477 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73900a8050 con 0x7f739c15be10 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f739be00640 1 -- 172.21.15.90:0/751358477 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900ac860 con 0x7f739c15be10 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 -- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 msgr2=0x7f739c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 --2- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c06ea10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7390009870 tx=0x7f739001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 -- 172.21.15.90:0/751358477 shutdown_connections 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 --2- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 --2- 172.21.15.90:0/751358477 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c06ea10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 -- 172.21.15.90:0/751358477 >> 172.21.15.90:0/751358477 conn(0x7f739c0b1880 msgr2=0x7f739c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 -- 172.21.15.90:0/751358477 shutdown_connections 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.128+0000 7f73a4600640 1 -- 172.21.15.90:0/751358477 wait complete. 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 Processor -- start 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- start start 2024-03-31T15:56:17.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c15a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f739c169160 con 0x7f739c0cb630 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f739c1692e0 con 0x7f739c15be10 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f739c169460 con 0x7f739c0cbae0 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48314/0 (socket says 172.21.15.90:48314) 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 -- 172.21.15.90:0/2881813836 learned_addr learned my addr 172.21.15.90:0/2881813836 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a2600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c15a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1200640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 -- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 msgr2=0x7f739c0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c0ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 -- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 msgr2=0x7f739c15a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c15a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 -- 172.21.15.90:0/2881813836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c151cb0 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a1c00640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f739001a100 tx=0x7f739001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900b1040 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73900bc070 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900b7890 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c15b9a0 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.132+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c15b3e0 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c0686d0 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f73900b7a30 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f7399600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5730+0+0 (secure 0 0 0) 0x7f7390120c10 con 0x7f739c0cb630 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f73a1200640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.136+0000 7f73a1200640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 0x7f737c06a840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f739c1584d0 tx=0x7f7388009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:17.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.140+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7390016150 con 0x7f739c0cb630 2024-03-31T15:56:17.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.300+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7f739c071570 con 0x7f739c0cb630 2024-03-31T15:56:17.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.964+0000 7f7399600640 1 -- 172.21.15.90:0/2881813836 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v47) ==== 160+0+0 (secure 0 0 0) 0x7f73900ace20 con 0x7f739c0cb630 2024-03-31T15:56:17.958 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'data_new' removed 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.964+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 msgr2=0x7f737c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.964+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 0x7f737c06a840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f739c1584d0 tx=0x7f7388009040 comp rx=0 tx=0).stop 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.964+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 msgr2=0x7f739c151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.964+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f739001a100 tx=0x7f739001ad10 comp rx=0 tx=0).stop 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 shutdown_connections 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f737c068380 0x7f737c06a840 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f739c15be10 0x7f739c15a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f739c0cbae0 0x7f739c0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 --2- 172.21.15.90:0/2881813836 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f739c0cb630 0x7f739c151770 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:17.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 >> 172.21.15.90:0/2881813836 conn(0x7f739c0b1880 msgr2=0x7f739c0af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:17.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 shutdown_connections 2024-03-31T15:56:17.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:17.968+0000 7f73a4600640 1 -- 172.21.15.90:0/2881813836 wait complete. 2024-03-31T15:56:17.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:56:18.009 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:56:18.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:56:18.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.124+0000 7f0d6a800640 1 Processor -- start 2024-03-31T15:56:18.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.124+0000 7f0d6a800640 1 -- start start 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d6a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d6415b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d6a800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d64072a60 con 0x7f0d640725a0 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d6a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d640aeac0 con 0x7f0d6415b040 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d6a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d640ae890 con 0x7f0d64071940 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d5be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0d640725a0 0x7f0d640b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39548/0 (socket says 172.21.15.90:39548) 2024-03-31T15:56:18.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d5be00640 1 -- 172.21.15.90:0/4085209273 learned_addr learned my addr 172.21.15.90:0/4085209273 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d6415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794382447 0 0) 0x7f0d64072a60 con 0x7f0d640725a0 2024-03-31T15:56:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d48005740 con 0x7f0d640725a0 2024-03-31T15:56:18.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905106901 0 0) 0x7f0d640aeac0 con 0x7f0d6415b040 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d64072a60 con 0x7f0d6415b040 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1876287538 0 0) 0x7f0d48005740 con 0x7f0d640725a0 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 msgr2=0x7f0d6415b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d6415b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 >> v1:172.21.15.102:6789/0 conn(0x7f0d6415b040 legacy=0x7f0d64071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d640aeda0 con 0x7f0d640725a0 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f0d4c001680 con 0x7f0d640725a0 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 >> v1:172.21.15.90:6789/0 conn(0x7f0d640725a0 legacy=0x7f0d640b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d48005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48006340 0x7f0d48006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d48006f60 0x7f0d480073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4800dcf0 con 0x7f0d48006f60 2024-03-31T15:56:18.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4800dea0 con 0x7f0d48006340 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4800e0a0 con 0x7f0d64071940 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d48006f60 0x7f0d480073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63400640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d48005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d5be00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48006340 0x7f0d48006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 -- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 msgr2=0x7f0d48005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d48005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 -- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48006340 msgr2=0x7f0d48006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48006340 0x7f0d48006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63e00640 1 -- 172.21.15.90:0/4085209273 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d48005740 con 0x7f0d48006f60 2024-03-31T15:56:18.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.128+0000 7f0d63400640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d64071940 0x7f0d48005d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:18.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d48006f60 0x7f0d480073e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009fc0 tx=0x7f0d5c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d46ff29ab1ba34 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0b2070 con 0x7f0d48006f60 2024-03-31T15:56:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0a5c90 con 0x7f0d48006f60 2024-03-31T15:56:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d62a00640 1 -- 172.21.15.90:0/4085209273 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c009c10 con 0x7f0d48006f60 2024-03-31T15:56:18.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 -- 172.21.15.90:0/4085209273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d48006f60 msgr2=0x7f0d480073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4085209273 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d48006f60 0x7f0d480073e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009fc0 tx=0x7f0d5c01cc80 comp rx=0 tx=0).stop 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d63e00640 1 -- 172.21.15.90:0/4085209273 reap_dead start 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 -- 172.21.15.90:0/4085209273 shutdown_connections 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 -- 172.21.15.90:0/4085209273 >> 172.21.15.90:0/4085209273 conn(0x7f0d640b18b0 msgr2=0x7f0d640b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 -- 172.21.15.90:0/4085209273 shutdown_connections 2024-03-31T15:56:18.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.132+0000 7f0d6a800640 1 -- 172.21.15.90:0/4085209273 wait complete. 2024-03-31T15:56:18.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 Processor -- start 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- start start 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d48006f60 0x7f0d641e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d64151a20 0x7f0d6414f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d640b35c0 con 0x7f0d641f3e60 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d64072d40 con 0x7f0d64151a20 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d640713f0 con 0x7f0d48006f60 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48340/0 (socket says 172.21.15.90:48340) 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 -- 172.21.15.90:0/4014972583 learned_addr learned my addr 172.21.15.90:0/4014972583 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:18.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d63400640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d48006f60 0x7f0d641e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d63e00640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d64151a20 0x7f0d6414f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 -- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d48006f60 msgr2=0x7f0d641e78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d48006f60 0x7f0d641e78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 -- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d64151a20 msgr2=0x7f0d6414f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d64151a20 0x7f0d6414f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 -- 172.21.15.90:0/4014972583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d641527d0 con 0x7f0d641f3e60 2024-03-31T15:56:18.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5be00640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0119b0 tx=0x7f0d4c011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c010040 con 0x7f0d641f3e60 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d4c01a070 con 0x7f0d641f3e60 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00fe20 con 0x7f0d641f3e60 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d641e7e30 con 0x7f0d641f3e60 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d64152210 con 0x7f0d641f3e60 2024-03-31T15:56:18.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.136+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d640685f0 con 0x7f0d641f3e60 2024-03-31T15:56:18.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.140+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d4c01f460 con 0x7f0d641f3e60 2024-03-31T15:56:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.140+0000 7f0d5b400640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 0x7f0d4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.140+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f0d4c089bc0 con 0x7f0d641f3e60 2024-03-31T15:56:18.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.140+0000 7f0d63400640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 0x7f0d4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.144+0000 7f0d63400640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 0x7f0d4006a840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007af0 tx=0x7f0d54007a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.144+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d4c05a250 con 0x7f0d641f3e60 2024-03-31T15:56:18.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.308+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0d64157cf0 con 0x7f0d641f3e60 2024-03-31T15:56:18.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d5b400640 1 -- 172.21.15.90:0/4014972583 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f0d4c059b50 con 0x7f0d641f3e60 2024-03-31T15:56:18.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 msgr2=0x7f0d4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 0x7f0d4006a840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007af0 tx=0x7f0d54007a60 comp rx=0 tx=0).stop 2024-03-31T15:56:18.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 msgr2=0x7f0d641514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0119b0 tx=0x7f0d4c011db0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 shutdown_connections 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d40068380 0x7f0d4006a840 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d64151a20 0x7f0d6414f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d641f3e60 0x7f0d641514e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 --2- 172.21.15.90:0/4014972583 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d48006f60 0x7f0d641e78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 >> 172.21.15.90:0/4014972583 conn(0x7f0d640b18b0 msgr2=0x7f0d640ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 shutdown_connections 2024-03-31T15:56:18.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.312+0000 7f0d6a800640 1 -- 172.21.15.90:0/4014972583 wait complete. 2024-03-31T15:56:18.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-31T15:56:18.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 Processor -- start 2024-03-31T15:56:18.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 -- start start 2024-03-31T15:56:18.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e1c1449a0 0x7f4e1c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e1c06bf30 0x7f4e1c15ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e1c0b47e0 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4e1c0b4ac0 con 0x7f4e1c06bf30 2024-03-31T15:56:18.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e23e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4e1c15b7d0 con 0x7f4e1c1449a0 2024-03-31T15:56:18.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e21e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e1c06bf30 0x7f4e1c15ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e21e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4e1c06bf30 0x7f4e1c15ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48348/0 (socket says 172.21.15.90:48348) 2024-03-31T15:56:18.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.508+0000 7f4e21e00640 1 -- 172.21.15.90:0/2677625970 learned_addr learned my addr 172.21.15.90:0/2677625970 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21400640 1 --2- 172.21.15.90:0/2677625970 >> v2:172.21.15.102:3300/0 conn(0x7f4e1c1449a0 0x7f4e1c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21e00640 1 -- 172.21.15.90:0/2677625970 >> v2:172.21.15.102:3300/0 conn(0x7f4e1c1449a0 msgr2=0x7f4e1c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21e00640 1 --2- 172.21.15.90:0/2677625970 >> v2:172.21.15.102:3300/0 conn(0x7f4e1c1449a0 0x7f4e1c15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21e00640 1 -- 172.21.15.90:0/2677625970 >> v1:172.21.15.102:6789/0 conn(0x7f4e1c0b3d90 legacy=0x7f4e1c06b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:18.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21e00640 1 -- 172.21.15.90:0/2677625970 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1c15bab0 con 0x7f4e1c06bf30 2024-03-31T15:56:18.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e21e00640 1 --2- 172.21.15.90:0/2677625970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c15ea50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0b4e60 tx=0x7f4e1401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34968f243ec5c74b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e1be00640 1 -- 172.21.15.90:0/2677625970 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140a5780 con 0x7f4e1c06bf30 2024-03-31T15:56:18.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e1be00640 1 -- 172.21.15.90:0/2677625970 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140a5920 con 0x7f4e1c06bf30 2024-03-31T15:56:18.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e1be00640 1 -- 172.21.15.90:0/2677625970 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140a5cf0 con 0x7f4e1c06bf30 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e23e00640 1 -- 172.21.15.90:0/2677625970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c06bf30 msgr2=0x7f4e1c15ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.512+0000 7f4e23e00640 1 --2- 172.21.15.90:0/2677625970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c15ea50 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0b4e60 tx=0x7f4e1401c9b0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- 172.21.15.90:0/2677625970 shutdown_connections 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 --2- 172.21.15.90:0/2677625970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c15ea50 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 --2- 172.21.15.90:0/2677625970 >> v2:172.21.15.102:3300/0 conn(0x7f4e1c1449a0 0x7f4e1c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- 172.21.15.90:0/2677625970 >> 172.21.15.90:0/2677625970 conn(0x7f4e1c0b1c80 msgr2=0x7f4e1c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- 172.21.15.90:0/2677625970 shutdown_connections 2024-03-31T15:56:18.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- 172.21.15.90:0/2677625970 wait complete. 2024-03-31T15:56:18.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 Processor -- start 2024-03-31T15:56:18.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- start start 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e1c1449a0 0x7f4e1c1f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1c168180 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1c168000 con 0x7f4e1c06bf30 2024-03-31T15:56:18.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e23e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4e1c168300 con 0x7f4e1c1449a0 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48350/0 (socket says 172.21.15.90:48350) 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 -- 172.21.15.90:0/3200612046 learned_addr learned my addr 172.21.15.90:0/3200612046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e21400640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e21e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e1c1449a0 0x7f4e1c1f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 -- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e1c1449a0 msgr2=0x7f4e1c1f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e1c1449a0 0x7f4e1c1f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 -- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e1c06bf30 msgr2=0x7f4e1c153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 -- 172.21.15.90:0/3200612046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1c13e750 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e20a00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c450 tx=0x7f4e0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.516+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08012830 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e08017070 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08012c30 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e1c13ea30 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e1c13eff0 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e1c0686d0 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.515 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e0801c460 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e19600640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 0x7f4dfc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:18.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f4e080864a0 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e21400640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 0x7f4dfc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:18.516 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.520+0000 7f4e21400640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 0x7f4dfc06a840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c1f8850 tx=0x7f4e10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:18.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.524+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4e08056580 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.732+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f4e1c13e010 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.964+0000 7f4e19600640 1 -- 172.21.15.90:0/3200612046 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v157) ==== 147+0+0 (secure 0 0 0) 0x7f4e0805a880 con 0x7f4e1c0b3d90 2024-03-31T15:56:18.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 msgr2=0x7f4dfc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 0x7f4dfc06a840 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c1f8850 tx=0x7f4e10008040 comp rx=0 tx=0).stop 2024-03-31T15:56:18.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 msgr2=0x7f4e1c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800c450 tx=0x7f4e0800c910 comp rx=0 tx=0).stop 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 shutdown_connections 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4dfc068380 0x7f4dfc06a840 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4e1c1449a0 0x7f4e1c1f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4e1c0b3d90 0x7f4e1c0cfac0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 --2- 172.21.15.90:0/3200612046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e1c06bf30 0x7f4e1c153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 >> 172.21.15.90:0/3200612046 conn(0x7f4e1c0b1c80 msgr2=0x7f4e1c0d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 shutdown_connections 2024-03-31T15:56:18.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:18.968+0000 7f4e23e00640 1 -- 172.21.15.90:0/3200612046 wait complete. 2024-03-31T15:56:18.981 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-31T15:56:18.982 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:56:18.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:19.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 Processor -- start 2024-03-31T15:56:19.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 -- start start 2024-03-31T15:56:19.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f532806c7b0 0x7f532806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f532806d5f0 0x7f532815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f532815f780 con 0x7f532815e1b0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f532806daa0 con 0x7f532806d5f0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5330600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f53281622b0 con 0x7f532806c7b0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532dc00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f532815e1b0 0x7f5328162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39552/0 (socket says 172.21.15.90:39552) 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532dc00640 1 -- 172.21.15.90:0/316270193 learned_addr learned my addr 172.21.15.90:0/316270193 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532d200640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f532806c7b0 0x7f532806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532e600640 1 --2- 172.21.15.90:0/316270193 >> v2:172.21.15.102:3300/0 conn(0x7f532806d5f0 0x7f532815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3619302558 0 0) 0x7f532815f780 con 0x7f532815e1b0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5310003530 con 0x7f532815e1b0 2024-03-31T15:56:19.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 821336697 0 0) 0x7f5310003530 con 0x7f532815e1b0 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f532806c7b0 msgr2=0x7f532806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f532806c7b0 0x7f532806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 >> v2:172.21.15.102:3300/0 conn(0x7f532806d5f0 msgr2=0x7f532815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 --2- 172.21.15.90:0/316270193 >> v2:172.21.15.102:3300/0 conn(0x7f532806d5f0 0x7f532815e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532815f1c0 con 0x7f532815e1b0 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f531c001680 con 0x7f532815e1b0 2024-03-31T15:56:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532e600640 1 --2- 172.21.15.90:0/316270193 >> v2:172.21.15.102:3300/0 conn(0x7f532806d5f0 0x7f532815e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 >> v1:172.21.15.90:6789/0 conn(0x7f532815e1b0 legacy=0x7f5328162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53100039c0 0x7f5310003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310004810 0x7f5310004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53100051d0 0x7f531000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f532806daa0 con 0x7f53100039c0 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f53281622b0 con 0x7f5310004810 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f531000bf40 con 0x7f53100051d0 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.124+0000 7f532d200640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f532806c7b0 0x7f532806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53100039c0 0x7f5310003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532e600640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53100051d0 0x7f531000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532d200640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310004810 0x7f5310004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 -- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53100051d0 msgr2=0x7f531000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53100051d0 0x7f531000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 -- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310004810 msgr2=0x7f5310004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310004810 0x7f5310004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 -- 172.21.15.90:0/316270193 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310003530 con 0x7f53100039c0 2024-03-31T15:56:19.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532d200640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310004810 0x7f5310004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f532dc00640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53100039c0 0x7f5310003de0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c240 tx=0x7f531c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc9cb28cb4b1302 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c007d60 con 0x7f53100039c0 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f531c0a9020 con 0x7f53100039c0 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.128+0000 7f5327e00640 1 -- 172.21.15.90:0/316270193 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c005ae0 con 0x7f53100039c0 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- 172.21.15.90:0/316270193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53100039c0 msgr2=0x7f5310003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 --2- 172.21.15.90:0/316270193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53100039c0 0x7f5310003de0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c240 tx=0x7f531c01cc30 comp rx=0 tx=0).stop 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 -- 172.21.15.90:0/316270193 reap_dead start 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- 172.21.15.90:0/316270193 shutdown_connections 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- 172.21.15.90:0/316270193 >> 172.21.15.90:0/316270193 conn(0x7f532809aa40 msgr2=0x7f532809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- 172.21.15.90:0/316270193 shutdown_connections 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- 172.21.15.90:0/316270193 wait complete. 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 Processor -- start 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- start start 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 0x7f5328157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 0x7f532809ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53280d2940 con 0x7f5328162f20 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f53280d27c0 con 0x7f53100039c0 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f5330600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f53280d2ac0 con 0x7f53281638b0 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48370/0 (socket says 172.21.15.90:48370) 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 -- 172.21.15.90:0/2801228636 learned_addr learned my addr 172.21.15.90:0/2801228636 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532d200640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 0x7f532809ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 -- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 msgr2=0x7f532809ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532dc00640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 0x7f5328157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 0x7f532809ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 -- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 msgr2=0x7f5328157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 0x7f5328157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 -- 172.21.15.90:0/2801228636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53281f89d0 con 0x7f5328162f20 2024-03-31T15:56:19.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532d200640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 0x7f532809ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532dc00640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 0x7f5328157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.132+0000 7f532e600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f53280b3710 tx=0x7f532000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532000fdc0 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53200039a0 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53281f7c10 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320003d90 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53281f81d0 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53280686d0 con 0x7f5328162f20 2024-03-31T15:56:19.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f532001f020 con 0x7f5328162f20 2024-03-31T15:56:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 0x7f530806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f5320086c60 con 0x7f5328162f20 2024-03-31T15:56:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f532dc00640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 0x7f530806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.136+0000 7f532dc00640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 0x7f530806a840 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f531c0aa000 tx=0x7f531c001ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.140+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5320010040 con 0x7f5328162f20 2024-03-31T15:56:19.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.296+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f532806fad0 con 0x7f5328162f20 2024-03-31T15:56:19.293 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5325600640 1 -- 172.21.15.90:0/2801228636 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+13255 (secure 0 0 0) 0x7f53200572f0 con 0x7f5328162f20 2024-03-31T15:56:19.293 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:19.293 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":47,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:17.961897+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6834/353199836":"2024-04-01T15:56:13.945123+0000","172.21.15.90:6832/353199836":"2024-04-01T15:56:13.945123+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 msgr2=0x7f530806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 0x7f530806a840 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f531c0aa000 tx=0x7f531c001ac0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 msgr2=0x7f532809b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f53280b3710 tx=0x7f532000f580 comp rx=0 tx=0).stop 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 shutdown_connections 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5308068380 0x7f530806a840 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5328162f20 0x7f532809b390 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f53281638b0 0x7f532809ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 --2- 172.21.15.90:0/2801228636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53100039c0 0x7f5328157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 >> 172.21.15.90:0/2801228636 conn(0x7f532809aa40 msgr2=0x7f532806fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 shutdown_connections 2024-03-31T15:56:19.296 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.300+0000 7f5330600640 1 -- 172.21.15.90:0/2801228636 wait complete. 2024-03-31T15:56:19.314 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:19.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:19.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.456+0000 7fc6ea600640 1 Processor -- start 2024-03-31T15:56:19.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 -- start start 2024-03-31T15:56:19.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e415c290 0x7fc6e40ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc6e40af210 0x7fc6e4071750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e406e9c0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc6e406de40 con 0x7fc6e40af210 2024-03-31T15:56:19.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6ea600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e406e120 con 0x7fc6e415c290 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6daa00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc6e40ad3d0 0x7fc6e40aeb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39554/0 (socket says 172.21.15.90:39554) 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6daa00640 1 -- 172.21.15.90:0/131820994 learned_addr learned my addr 172.21.15.90:0/131820994 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3e00640 1 --2- 172.21.15.90:0/131820994 >> v2:172.21.15.90:3300/0 conn(0x7fc6e40af210 0x7fc6e4071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3400640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e415c290 0x7fc6e40ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3e00640 1 -- 172.21.15.90:0/131820994 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e415c290 msgr2=0x7fc6e40ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3e00640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e415c290 0x7fc6e40ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3e00640 1 -- 172.21.15.90:0/131820994 >> v1:172.21.15.90:6789/0 conn(0x7fc6e40ad3d0 legacy=0x7fc6e40aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e3e00640 1 -- 172.21.15.90:0/131820994 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e406e400 con 0x7fc6e40af210 2024-03-31T15:56:19.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.460+0000 7fc6e2a00640 1 -- 172.21.15.90:0/131820994 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2307335280 0 0) 0x7fc6e406e9c0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6e3e00640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40af210 0x7fc6e4071750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009d90 tx=0x7fc6dc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2174fa81080111e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6e2a00640 1 -- 172.21.15.90:0/131820994 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0b2070 con 0x7fc6e40af210 2024-03-31T15:56:19.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6e2a00640 1 -- 172.21.15.90:0/131820994 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc0a5830 con 0x7fc6e40af210 2024-03-31T15:56:19.457 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6e2a00640 1 -- 172.21.15.90:0/131820994 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0a5c00 con 0x7fc6e40af210 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- 172.21.15.90:0/131820994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40af210 msgr2=0x7fc6e4071750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40af210 0x7fc6e4071750 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc009d90 tx=0x7fc6dc01c970 comp rx=0 tx=0).stop 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- 172.21.15.90:0/131820994 shutdown_connections 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40af210 0x7fc6e4071750 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 --2- 172.21.15.90:0/131820994 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e415c290 0x7fc6e40ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- 172.21.15.90:0/131820994 >> 172.21.15.90:0/131820994 conn(0x7fc6e40b18b0 msgr2=0x7fc6e40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- 172.21.15.90:0/131820994 shutdown_connections 2024-03-31T15:56:19.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- 172.21.15.90:0/131820994 wait complete. 2024-03-31T15:56:19.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 Processor -- start 2024-03-31T15:56:19.459 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.464+0000 7fc6ea600640 1 -- start start 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e40af210 0x7fc6e4155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6e415c290 0x7fc6e4156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e4072870 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e40726f0 con 0x7fc6e415c290 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e40729f0 con 0x7fc6e40af210 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48398/0 (socket says 172.21.15.90:48398) 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 -- 172.21.15.90:0/235810045 learned_addr learned my addr 172.21.15.90:0/235810045 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:19.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 -- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e40af210 msgr2=0x7fc6e4155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e40af210 0x7fc6e4155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 -- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6e415c290 msgr2=0x7fc6e4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6e415c290 0x7fc6e4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 -- 172.21.15.90:0/235810045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e41594a0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6e3400640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e870 tx=0x7fc6d400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4017070 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d400c940 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d400ccf0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e4159780 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e409abe0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e4075880 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.468+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc6d401a070 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.472+0000 7fc6dbe00640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 0x7fc6c006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.472+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7fc6d4086c00 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.472+0000 7fc6daa00640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 0x7fc6c006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.472+0000 7fc6daa00640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 0x7fc6c006a840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0011f0 tx=0x7fc6cc007930 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.476+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc6d4057200 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.628+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc6e4159ca0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.628+0000 7fc6dbe00640 1 -- 172.21.15.90:0/235810045 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7fc6e4159ca0 con 0x7fc6e40ad3d0 2024-03-31T15:56:19.624 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:19.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 msgr2=0x7fc6c006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 0x7fc6c006a840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0011f0 tx=0x7fc6cc007930 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 msgr2=0x7fc6e4146740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc6d400e870 tx=0x7fc6d400ec70 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 shutdown_connections 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc6c0068380 0x7fc6c006a840 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6e415c290 0x7fc6e4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc6e40af210 0x7fc6e4155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 --2- 172.21.15.90:0/235810045 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6e40ad3d0 0x7fc6e4146740 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 >> 172.21.15.90:0/235810045 conn(0x7fc6e40b18b0 msgr2=0x7fc6e415b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 shutdown_connections 2024-03-31T15:56:19.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.632+0000 7fc6ea600640 1 -- 172.21.15.90:0/235810045 wait complete. 2024-03-31T15:56:19.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-31T15:56:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.784+0000 7f5c0f200640 1 Processor -- start 2024-03-31T15:56:19.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.784+0000 7f5c0f200640 1 -- start start 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0f200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5c1006f370 0x7f5c1015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0f200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 0x7f5c10160790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1006f800 con 0x7f5c100d08e0 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0f200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5c1006fac0 con 0x7f5c1006f370 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0f200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1006fda0 con 0x7f5c10160300 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5c1006f370 0x7f5c1015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5c1006f370 0x7f5c1015fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48404/0 (socket says 172.21.15.90:48404) 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 -- 172.21.15.90:0/2813052404 learned_addr learned my addr 172.21.15.90:0/2813052404 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0e800640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 0x7f5c10160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 -- 172.21.15.90:0/2813052404 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 msgr2=0x7f5c10160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 0x7f5c10160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 -- 172.21.15.90:0/2813052404 >> v1:172.21.15.102:6789/0 conn(0x7f5c100d08e0 legacy=0x7f5c100d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:19.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 -- 172.21.15.90:0/2813052404 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c10070080 con 0x7f5c1006f370 2024-03-31T15:56:19.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0e800640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 0x7f5c10160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:19.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c05400640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c1006f370 0x7f5c1015fdc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80098a0 tx=0x7f5bf801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef2cb78504f3462 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0d400640 1 -- 172.21.15.90:0/2813052404 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80b0070 con 0x7f5c1006f370 2024-03-31T15:56:19.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0d400640 1 -- 172.21.15.90:0/2813052404 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bf80ab600 con 0x7f5c1006f370 2024-03-31T15:56:19.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.788+0000 7f5c0d400640 1 -- 172.21.15.90:0/2813052404 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80ab9d0 con 0x7f5c1006f370 2024-03-31T15:56:19.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- 172.21.15.90:0/2813052404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c1006f370 msgr2=0x7f5c1015fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c1006f370 0x7f5c1015fdc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80098a0 tx=0x7f5bf801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- 172.21.15.90:0/2813052404 shutdown_connections 2024-03-31T15:56:19.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c10160300 0x7f5c10160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- 172.21.15.90:0/2813052404 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c1006f370 0x7f5c1015fdc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- 172.21.15.90:0/2813052404 >> 172.21.15.90:0/2813052404 conn(0x7f5c100d1d70 msgr2=0x7f5c100d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:19.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- 172.21.15.90:0/2813052404 shutdown_connections 2024-03-31T15:56:19.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- 172.21.15.90:0/2813052404 wait complete. 2024-03-31T15:56:19.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 Processor -- start 2024-03-31T15:56:19.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- start start 2024-03-31T15:56:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 0x7f5c10154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c100d08e0 0x7f5c1014a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1015e5c0 con 0x7f5c10160300 2024-03-31T15:56:19.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1015e440 con 0x7f5c1006f370 2024-03-31T15:56:19.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0f200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5c1015e740 con 0x7f5c100d08e0 2024-03-31T15:56:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48418/0 (socket says 172.21.15.90:48418) 2024-03-31T15:56:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 -- 172.21.15.90:0/1689415777 learned_addr learned my addr 172.21.15.90:0/1689415777 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c05400640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c100d08e0 0x7f5c1014a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0de00640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 0x7f5c10154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 -- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c100d08e0 msgr2=0x7f5c1014a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c100d08e0 0x7f5c1014a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 -- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 msgr2=0x7f5c10154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 0x7f5c10154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0e800640 1 -- 172.21.15.90:0/1689415777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c100af7b0 con 0x7f5c10160300 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.792+0000 7f5c0de00640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 0x7f5c10154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c0e800640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c820 tx=0x7f5c0800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08015020 con 0x7f5c10160300 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c08004240 con 0x7f5c10160300 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08003620 con 0x7f5c10160300 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c100afa90 con 0x7f5c10160300 2024-03-31T15:56:19.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c100b0050 con 0x7f5c10160300 2024-03-31T15:56:19.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c100685f0 con 0x7f5c10160300 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c08018020 con 0x7f5c10160300 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 0x7f5bec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.796+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f5c08086820 con 0x7f5c10160300 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.800+0000 7f5c0de00640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 0x7f5bec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.800+0000 7f5c0de00640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 0x7f5bec06a840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5c101511f0 tx=0x7f5c00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:19.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:19.800+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5c0801f020 con 0x7f5c10160300 2024-03-31T15:56:20.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.012+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7f5c1006c2e0 con 0x7f5c10160300 2024-03-31T15:56:20.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.972+0000 7f5c06a00640 1 -- 172.21.15.90:0/1689415777 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v159) ==== 137+0+0 (secure 0 0 0) 0x7f5c08056f50 con 0x7f5c10160300 2024-03-31T15:56:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 msgr2=0x7f5bec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 0x7f5bec06a840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5c101511f0 tx=0x7f5c00009040 comp rx=0 tx=0).stop 2024-03-31T15:56:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 msgr2=0x7f5c1014adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:20.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800c820 tx=0x7f5c0800cce0 comp rx=0 tx=0).stop 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 shutdown_connections 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5bec068380 0x7f5bec06a840 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5c10160300 0x7f5c1014adf0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5c100d08e0 0x7f5c1014a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 --2- 172.21.15.90:0/1689415777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c1006f370 0x7f5c10154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 >> 172.21.15.90:0/1689415777 conn(0x7f5c100d1d70 msgr2=0x7f5c1006db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 shutdown_connections 2024-03-31T15:56:20.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:20.976+0000 7f5c0f200640 1 -- 172.21.15.90:0/1689415777 wait complete. 2024-03-31T15:56:20.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-31T15:56:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 Processor -- start 2024-03-31T15:56:21.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 -- start start 2024-03-31T15:56:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7b700aeb00 0x7f7b700aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7006e9c0 con 0x7f7b7015c290 2024-03-31T15:56:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7006de40 con 0x7f7b70071750 2024-03-31T15:56:21.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b78400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7006e120 con 0x7f7b700aeb00 2024-03-31T15:56:21.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b75a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7b7015c290 0x7f7b700ace90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39556/0 (socket says 172.21.15.90:39556) 2024-03-31T15:56:21.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b75a00640 1 -- 172.21.15.90:0/3248189615 learned_addr learned my addr 172.21.15.90:0/3248189615 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:21.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.140+0000 7f7b75000640 1 --2- 172.21.15.90:0/3248189615 >> v2:172.21.15.102:3300/0 conn(0x7f7b700aeb00 0x7f7b700aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962890405 0 0) 0x7f7b7006e9c0 con 0x7f7b7015c290 2024-03-31T15:56:21.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b58003530 con 0x7f7b7015c290 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 328002661 0 0) 0x7f7b7006de40 con 0x7f7b70071750 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b7006e9c0 con 0x7f7b70071750 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 988316818 0 0) 0x7f7b58003530 con 0x7f7b7015c290 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 >> v2:172.21.15.102:3300/0 conn(0x7f7b700aeb00 msgr2=0x7f7b700aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 --2- 172.21.15.90:0/3248189615 >> v2:172.21.15.102:3300/0 conn(0x7f7b700aeb00 0x7f7b700aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 >> v1:172.21.15.102:6789/0 conn(0x7f7b70071750 legacy=0x7f7b70071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7006e400 con 0x7f7b7015c290 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7b64001680 con 0x7f7b7015c290 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 >> v1:172.21.15.90:6789/0 conn(0x7f7b7015c290 legacy=0x7f7b700ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b58003b50 0x7f7b58003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b5800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7006e120 con 0x7f7b580051d0 2024-03-31T15:56:21.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5800c050 con 0x7f7b58003b50 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b5800c250 con 0x7f7b58004810 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b5800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b75a00640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b58003b50 0x7f7b58003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b75000640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 -- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 msgr2=0x7f7b58004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b58004810 0x7f7b58004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 -- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b58003b50 msgr2=0x7f7b58003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b58003b50 0x7f7b58003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 -- 172.21.15.90:0/3248189615 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b58003530 con 0x7f7b580051d0 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.144+0000 7f7b76400640 1 -- 172.21.15.90:0/3248189615 reap_dead start 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b76400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b5800b9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7b68001fd0 tx=0x7f7b6801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a338ba1535a13daf server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b68001c20 con 0x7f7b580051d0 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b68003b80 con 0x7f7b580051d0 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b6fe00640 1 -- 172.21.15.90:0/3248189615 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680050d0 con 0x7f7b580051d0 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 msgr2=0x7f7b5800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b5800b9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7b68001fd0 tx=0x7f7b6801c900 comp rx=0 tx=0).stop 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- 172.21.15.90:0/3248189615 shutdown_connections 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 --2- 172.21.15.90:0/3248189615 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b5800b9d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- 172.21.15.90:0/3248189615 >> 172.21.15.90:0/3248189615 conn(0x7f7b700b18b0 msgr2=0x7f7b700b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- 172.21.15.90:0/3248189615 shutdown_connections 2024-03-31T15:56:21.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- 172.21.15.90:0/3248189615 wait complete. 2024-03-31T15:56:21.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 Processor -- start 2024-03-31T15:56:21.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- start start 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 0x7f7b70153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b701f2ec0 0x7f7b701f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7006db70 con 0x7f7b580051d0 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7006d9f0 con 0x7f7b70155e50 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b78400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7b700aef80 con 0x7f7b701f2ec0 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b75a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b75a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48434/0 (socket says 172.21.15.90:48434) 2024-03-31T15:56:21.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b75a00640 1 -- 172.21.15.90:0/174859482 learned_addr learned my addr 172.21.15.90:0/174859482 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b76400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b701f2ec0 0x7f7b701f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.148+0000 7f7b75000640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 0x7f7b70153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 -- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b701f2ec0 msgr2=0x7f7b701f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b701f2ec0 0x7f7b701f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 -- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 msgr2=0x7f7b70153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 0x7f7b70153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 -- 172.21.15.90:0/174859482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b701f3880 con 0x7f7b580051d0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75000640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 0x7f7b70153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b75a00640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7b64012510 tx=0x7f7b640129d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6401c070 con 0x7f7b580051d0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b64014810 con 0x7f7b580051d0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b701f76b0 con 0x7f7b580051d0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64014c00 con 0x7f7b580051d0 2024-03-31T15:56:21.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b701f7c70 con 0x7f7b580051d0 2024-03-31T15:56:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b70075880 con 0x7f7b580051d0 2024-03-31T15:56:21.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7b6401f070 con 0x7f7b580051d0 2024-03-31T15:56:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.152+0000 7f7b6d600640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 0x7f7b5006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.156+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f7b64089180 con 0x7f7b580051d0 2024-03-31T15:56:21.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.156+0000 7f7b75000640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 0x7f7b5006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.156+0000 7f7b75000640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 0x7f7b5006a840 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b700ce960 tx=0x7f7b5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.160+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7b64059250 con 0x7f7b580051d0 2024-03-31T15:56:21.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.316+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7b700685f0 con 0x7f7b580051d0 2024-03-31T15:56:21.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.320+0000 7f7b6d600640 1 -- 172.21.15.90:0/174859482 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f7b6405d550 con 0x7f7b580051d0 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 msgr2=0x7f7b5006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 0x7f7b5006a840 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b700ce960 tx=0x7f7b5c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 msgr2=0x7f7b701f6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7b64012510 tx=0x7f7b640129d0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 shutdown_connections 2024-03-31T15:56:21.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7b50068380 0x7f7b5006a840 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7b701f2ec0 0x7f7b701f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b70155e50 0x7f7b70153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 --2- 172.21.15.90:0/174859482 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b580051d0 0x7f7b701f6fb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 >> 172.21.15.90:0/174859482 conn(0x7f7b700b18b0 msgr2=0x7f7b700af820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 shutdown_connections 2024-03-31T15:56:21.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.324+0000 7f7b78400640 1 -- 172.21.15.90:0/174859482 wait complete. 2024-03-31T15:56:21.336 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:56:21.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:21.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:21.444 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:56:21.444 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:21.444 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:21.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:56:21.453 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:21.454 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:21.462 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:21.462 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:21.463 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:21.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:56:21.503 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:21.504 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:21.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:21.513 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-31T15:56:21.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 Processor -- start 2024-03-31T15:56:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 -- start start 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fca2c070260 0x7fca2c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2c1556b0 con 0x7fca2c1552d0 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca2c0ae4a0 con 0x7fca2c0b4bc0 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca34600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fca2c0b41f0 con 0x7fca2c070260 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fca2c1552d0 0x7fca2c06b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:39562/0 (socket says 172.21.15.90:39562) 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31c00640 1 -- 172.21.15.90:0/2612195518 learned_addr learned my addr 172.21.15.90:0/2612195518 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31200640 1 --2- 172.21.15.90:0/2612195518 >> v2:172.21.15.90:3300/0 conn(0x7fca2c070260 0x7fca2c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca2be00640 1 -- 172.21.15.90:0/2612195518 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187961318 0 0) 0x7fca20001120 con 0x7fca2c1552d0 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca2be00640 1 -- 172.21.15.90:0/2612195518 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca14003530 con 0x7fca2c1552d0 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31200640 1 -- 172.21.15.90:0/2612195518 >> v1:172.21.15.102:6789/0 conn(0x7fca2c0b4bc0 legacy=0x7fca2c15e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31200640 1 -- 172.21.15.90:0/2612195518 >> v1:172.21.15.90:6789/0 conn(0x7fca2c1552d0 legacy=0x7fca2c06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.684 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.688+0000 7fca31200640 1 -- 172.21.15.90:0/2612195518 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c0ae780 con 0x7fca2c070260 2024-03-31T15:56:21.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca31200640 1 --2- 172.21.15.90:0/2612195518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c0b4680 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fca180097c0 tx=0x7fca1801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1473261e55caf6f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca2be00640 1 -- 172.21.15.90:0/2612195518 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca180b0070 con 0x7fca2c070260 2024-03-31T15:56:21.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca2be00640 1 -- 172.21.15.90:0/2612195518 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca180ab600 con 0x7fca2c070260 2024-03-31T15:56:21.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca2be00640 1 -- 172.21.15.90:0/2612195518 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca180ab9d0 con 0x7fca2c070260 2024-03-31T15:56:21.687 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 -- 172.21.15.90:0/2612195518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 msgr2=0x7fca2c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 --2- 172.21.15.90:0/2612195518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c0b4680 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fca180097c0 tx=0x7fca1801c460 comp rx=0 tx=0).stop 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 -- 172.21.15.90:0/2612195518 shutdown_connections 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 --2- 172.21.15.90:0/2612195518 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c0b4680 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 -- 172.21.15.90:0/2612195518 >> 172.21.15.90:0/2612195518 conn(0x7fca2c0d0650 msgr2=0x7fca2c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 -- 172.21.15.90:0/2612195518 shutdown_connections 2024-03-31T15:56:21.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.692+0000 7fca34600640 1 -- 172.21.15.90:0/2612195518 wait complete. 2024-03-31T15:56:21.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 Processor -- start 2024-03-31T15:56:21.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- start start 2024-03-31T15:56:21.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0b4bc0 0x7fca2c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 0x7fca2c15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c0b51f0 con 0x7fca2c070260 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c0b5070 con 0x7fca2c0b4bc0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fca2c16c110 con 0x7fca2c1552d0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48460/0 (socket says 172.21.15.90:48460) 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 -- 172.21.15.90:0/1081475085 learned_addr learned my addr 172.21.15.90:0/1081475085 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31200640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0b4bc0 0x7fca2c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca32600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 0x7fca2c15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 -- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 msgr2=0x7fca2c15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 0x7fca2c15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 -- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0b4bc0 msgr2=0x7fca2c06eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0b4bc0 0x7fca2c06eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 -- 172.21.15.90:0/1081475085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c068d60 con 0x7fca2c070260 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca32600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 0x7fca2c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca31c00640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c3c0 tx=0x7fca2000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20017070 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca20005cf0 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20007310 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2c1512c0 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.696+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2c151880 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca2c0686d0 con 0x7fca2c070260 2024-03-31T15:56:21.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fca2001a070 con 0x7fca2c070260 2024-03-31T15:56:21.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca29600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 0x7fca0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:21.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7fca20085b70 con 0x7fca2c070260 2024-03-31T15:56:21.694 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca31200640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 0x7fca0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:21.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.700+0000 7fca31200640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 0x7fca0c06a840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fca180ad000 tx=0x7fca180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:21.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.704+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fca2c1556b0 con 0x7fca2c070260 2024-03-31T15:56:21.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca2c0b3bc0 con 0x7fca2c070260 2024-03-31T15:56:21.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca29600640 1 -- 172.21.15.90:0/1081475085 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+3875 (secure 0 0 0) 0x7fca20055c40 con 0x7fca2c070260 2024-03-31T15:56:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:21.886 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5024,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/3569630718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":3569630718},{"type":"v1","addr":"172.21.15.90:6835","nonce":3569630718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5368,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2348161003","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2348161003},{"type":"v1","addr":"172.21.15.102:6836","nonce":2348161003}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1988216143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":1988216143},{"type":"v1","addr":"172.21.15.102:6837","nonce":1988216143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":6123,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/2879183365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":2879183365},{"type":"v1","addr":"172.21.15.90:6834","nonce":2879183365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[]} 2024-03-31T15:56:21.886 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 20 2024-03-31T15:56:21.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 msgr2=0x7fca0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 0x7fca0c06a840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fca180ad000 tx=0x7fca180a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 msgr2=0x7fca2c150c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c3c0 tx=0x7fca2000c880 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 shutdown_connections 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fca0c068380 0x7fca0c06a840 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fca2c1552d0 0x7fca2c15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca2c0b4bc0 0x7fca2c06eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 --2- 172.21.15.90:0/1081475085 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fca2c070260 0x7fca2c150c50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 >> 172.21.15.90:0/1081475085 conn(0x7fca2c0d0650 msgr2=0x7fca2c0b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 shutdown_connections 2024-03-31T15:56:21.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:21.892+0000 7fca34600640 1 -- 172.21.15.90:0/1081475085 wait complete. 2024-03-31T15:56:21.907 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-31T15:56:21.907 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-31T15:56:21.908 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:56:21.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:21.908 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-31T15:56:21.908 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-31T15:56:21.908 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:56:21.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:21.908 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-31T15:56:21.908 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-31T15:56:21.909 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:56:21.909 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:21.909 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-31T15:56:21.909 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-31T15:56:21.909 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:56:21.909 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:21.920 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:56:21.921 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-31T15:56:21.922 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:56:21.922 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-31T15:56:21.924 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:56:21.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-31T15:56:21.925 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:56:21.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-31T15:56:21.966 INFO:tasks.ceph.mds.a:Started 2024-03-31T15:56:21.967 DEBUG:teuthology.parallel:result is None 2024-03-31T15:56:21.968 INFO:tasks.ceph.mds.c:Started 2024-03-31T15:56:21.968 DEBUG:teuthology.parallel:result is None 2024-03-31T15:56:21.972 INFO:tasks.ceph.mds.b:Started 2024-03-31T15:56:21.972 DEBUG:teuthology.parallel:result is None 2024-03-31T15:56:21.974 INFO:tasks.ceph.mds.d:Started 2024-03-31T15:56:21.974 DEBUG:teuthology.parallel:result is None 2024-03-31T15:56:21.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-31T15:56:22.052 INFO:tasks.ceph.mds.c.smithi090.stdout:starting mds.c at 2024-03-31T15:56:22.053 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-31T15:56:22.060 INFO:tasks.ceph.mds.b.smithi102.stdout:starting mds.b at 2024-03-31T15:56:22.060 INFO:tasks.ceph.mds.d.smithi102.stdout:starting mds.d at 2024-03-31T15:56:22.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.124+0000 7f7a91600640 1 Processor -- start 2024-03-31T15:56:22.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.124+0000 7f7a91600640 1 -- start start 2024-03-31T15:56:22.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a8c15e320 0x7f7a8c0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c0d2940 0x7f7a8c161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7a8c1622d0 0x7f7a8c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8c15e700 con 0x7f7a8c15e320 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8c0d2dc0 con 0x7f7a8c1622d0 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a91600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c0d1400 con 0x7f7a8c0d2940 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a8c15e320 0x7f7a8c0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7a8c15e320 0x7f7a8c0d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48510/0 (socket says 172.21.15.90:48510) 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 -- 172.21.15.90:0/2945996833 learned_addr learned my addr 172.21.15.90:0/2945996833 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8b400640 1 --2- 172.21.15.90:0/2945996833 >> v2:172.21.15.102:3300/0 conn(0x7f7a8c1622d0 0x7f7a8c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8a000640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c0d2940 0x7f7a8c161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 -- 172.21.15.90:0/2945996833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c0d2940 msgr2=0x7f7a8c161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c0d2940 0x7f7a8c161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 -- 172.21.15.90:0/2945996833 >> v2:172.21.15.102:3300/0 conn(0x7f7a8c1622d0 msgr2=0x7f7a8c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:22.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/2945996833 >> v2:172.21.15.102:3300/0 conn(0x7f7a8c1622d0 0x7f7a8c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 -- 172.21.15.90:0/2945996833 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c0b1a20 con 0x7f7a8c15e320 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c15e320 0x7f7a8c0d0d20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0b7fb0 tx=0x7f7a7c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f158952017ee44f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a89600640 1 -- 172.21.15.90:0/2945996833 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0b0070 con 0x7f7a8c15e320 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a89600640 1 -- 172.21.15.90:0/2945996833 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7c0ab600 con 0x7f7a8c15e320 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.128+0000 7f7a89600640 1 -- 172.21.15.90:0/2945996833 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0ab9d0 con 0x7f7a8c15e320 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- 172.21.15.90:0/2945996833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c15e320 msgr2=0x7f7a8c0d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c15e320 0x7f7a8c0d0d20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0b7fb0 tx=0x7f7a7c01c260 comp rx=0 tx=0).stop 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- 172.21.15.90:0/2945996833 shutdown_connections 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- 172.21.15.90:0/2945996833 >> v2:172.21.15.102:3300/0 conn(0x7f7a8c1622d0 0x7f7a8c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c0d2940 0x7f7a8c161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- 172.21.15.90:0/2945996833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c15e320 0x7f7a8c0d0d20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- 172.21.15.90:0/2945996833 >> 172.21.15.90:0/2945996833 conn(0x7f7a8c08bfd0 msgr2=0x7f7a8c0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- 172.21.15.90:0/2945996833 shutdown_connections 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- 172.21.15.90:0/2945996833 wait complete. 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 Processor -- start 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- start start 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a8c0d2940 0x7f7a8c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c15e320 0x7f7a8c155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c0b2890 con 0x7f7a8c1622d0 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c0b2710 con 0x7f7a8c0d2940 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a91600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c0b2a10 con 0x7f7a8c15e320 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48512/0 (socket says 172.21.15.90:48512) 2024-03-31T15:56:22.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 -- 172.21.15.90:0/1874439593 learned_addr learned my addr 172.21.15.90:0/1874439593 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a8c0d2940 0x7f7a8c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8a000640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c15e320 0x7f7a8c155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 -- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c15e320 msgr2=0x7f7a8c155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c15e320 0x7f7a8c155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 -- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a8c0d2940 msgr2=0x7f7a8c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a8c0d2940 0x7f7a8c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.132+0000 7f7a8b400640 1 -- 172.21.15.90:0/1874439593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c0ce180 con 0x7f7a8c1622d0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a8b400640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400efc0 tx=0x7f7a8400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84018020 con 0x7f7a8c1622d0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a84014480 con 0x7f7a8c1622d0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84014870 con 0x7f7a8c1622d0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c0b6220 con 0x7f7a8c1622d0 2024-03-31T15:56:22.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c0b67e0 con 0x7f7a8c1622d0 2024-03-31T15:56:22.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c068660 con 0x7f7a8c1622d0 2024-03-31T15:56:22.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7a8401b020 con 0x7f7a8c1622d0 2024-03-31T15:56:22.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 0x7f7a6c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:22.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5335+0+0 (secure 0 0 0) 0x7f7a84087110 con 0x7f7a8c1622d0 2024-03-31T15:56:22.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.136+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 0x7f7a6c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:22.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.140+0000 7f7a8aa00640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 0x7f7a6c06a870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0ad000 tx=0x7f7a7c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:22.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.140+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7a84022070 con 0x7f7a8c1622d0 2024-03-31T15:56:22.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.308+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7a8c0c8af0 con 0x7f7a8c1622d0 2024-03-31T15:56:22.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.988+0000 7f7a82a00640 1 -- 172.21.15.90:0/1874439593 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v49) ==== 97+0+0 (secure 0 0 0) 0x7f7a84010040 con 0x7f7a8c1622d0 2024-03-31T15:56:22.982 INFO:teuthology.orchestra.run.smithi090.stderr: removed all blocklist entries 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 msgr2=0x7f7a6c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 0x7f7a6c06a870 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0ad000 tx=0x7f7a7c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 msgr2=0x7f7a8c1564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400efc0 tx=0x7f7a8400c490 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 shutdown_connections 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7a6c0683b0 0x7f7a6c06a870 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7a8c1622d0 0x7f7a8c1564f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7a8c15e320 0x7f7a8c155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 --2- 172.21.15.90:0/1874439593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a8c0d2940 0x7f7a8c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:22.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 >> 172.21.15.90:0/1874439593 conn(0x7f7a8c08bfd0 msgr2=0x7f7a8c15d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:22.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 shutdown_connections 2024-03-31T15:56:22.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:22.992+0000 7f7a91600640 1 -- 172.21.15.90:0/1874439593 wait complete. 2024-03-31T15:56:23.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-31T15:56:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.152+0000 7f7c23800640 1 Processor -- start 2024-03-31T15:56:23.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 -- start start 2024-03-31T15:56:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c1c0b3d90 0x7f7c1c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c1c0b47e0 con 0x7f7c1c06b820 2024-03-31T15:56:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7c1c0b4ac0 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c23800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1c15b7d0 con 0x7f7c1c1449a0 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c20e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c21800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7c1c06b820 0x7f7c1c15ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:53550/0 (socket says 172.21.15.90:53550) 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c13e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c1c0b3d90 0x7f7c1c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c20e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:37974/0 (socket says 172.21.15.90:37974) 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c21800640 1 -- 172.21.15.90:0/3993313268 learned_addr learned my addr 172.21.15.90:0/3993313268 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c13e00640 1 -- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 msgr2=0x7f7c1c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c13e00640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c13e00640 1 -- 172.21.15.90:0/3993313268 >> v1:172.21.15.102:6789/0 conn(0x7f7c1c06b820 legacy=0x7f7c1c15ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c13e00640 1 -- 172.21.15.90:0/3993313268 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c15bab0 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.156+0000 7f7c20e00640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:23.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c13e00640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c0b3d90 0x7f7c1c0b41f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7c040097c0 tx=0x7f7c1c15f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d84beb90df80b945 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c1be00640 1 -- 172.21.15.90:0/3993313268 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040a86a0 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c1be00640 1 -- 172.21.15.90:0/3993313268 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c040a8840 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c1be00640 1 -- 172.21.15.90:0/3993313268 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040a8c10 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 -- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c0b3d90 msgr2=0x7f7c1c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c0b3d90 0x7f7c1c0b41f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7c040097c0 tx=0x7f7c1c15f970 comp rx=0 tx=0).stop 2024-03-31T15:56:23.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 -- 172.21.15.90:0/3993313268 shutdown_connections 2024-03-31T15:56:23.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c0b3d90 0x7f7c1c0b41f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 --2- 172.21.15.90:0/3993313268 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c1449a0 0x7f7c1c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 -- 172.21.15.90:0/3993313268 >> 172.21.15.90:0/3993313268 conn(0x7f7c1c0b1c80 msgr2=0x7f7c1c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:23.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 -- 172.21.15.90:0/3993313268 shutdown_connections 2024-03-31T15:56:23.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.160+0000 7f7c23800640 1 -- 172.21.15.90:0/3993313268 wait complete. 2024-03-31T15:56:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 Processor -- start 2024-03-31T15:56:23.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- start start 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c0b3d90 0x7f7c1c163f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c1449a0 0x7f7c1c0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c0b2e00 con 0x7f7c1c06b820 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c0b2c80 con 0x7f7c1c1449a0 2024-03-31T15:56:23.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1c0b2f80 con 0x7f7c1c0b3d90 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41702/0 (socket says 172.21.15.90:41702) 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 -- 172.21.15.90:0/962069439 learned_addr learned my addr 172.21.15.90:0/962069439 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c21800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c1449a0 0x7f7c1c0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c13e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c0b3d90 0x7f7c1c163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 -- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c0b3d90 msgr2=0x7f7c1c163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c0b3d90 0x7f7c1c163f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 -- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c1449a0 msgr2=0x7f7c1c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c1449a0 0x7f7c1c0cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 -- 172.21.15.90:0/962069439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c164930 con 0x7f7c1c06b820 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c20e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e980 tx=0x7f7c0c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c017070 con 0x7f7c1c06b820 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0c00ca00 con 0x7f7c1c06b820 2024-03-31T15:56:23.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c164c10 con 0x7f7c1c06b820 2024-03-31T15:56:23.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00cdd0 con 0x7f7c1c06b820 2024-03-31T15:56:23.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1651d0 con 0x7f7c1c06b820 2024-03-31T15:56:23.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.164+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c0686d0 con 0x7f7c1c06b820 2024-03-31T15:56:23.160 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.168+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7c0c01a070 con 0x7f7c1c06b820 2024-03-31T15:56:23.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.168+0000 7f7c19600640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 0x7f7bf806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.168+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5249+0+0 (secure 0 0 0) 0x7f7c0c086ab0 con 0x7f7c1c06b820 2024-03-31T15:56:23.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.168+0000 7f7c13e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 0x7f7bf806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.168+0000 7f7c13e00640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 0x7f7bf806a840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7c040af000 tx=0x7f7c040ab040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.165 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.172+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7c0c056b40 con 0x7f7c1c06b820 2024-03-31T15:56:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7c1c072050 con 0x7f7c1c06b820 2024-03-31T15:56:23.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c19600640 1 -- 172.21.15.90:0/962069439 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5298 (secure 0 0 0) 0x7f7c0c05ae40 con 0x7f7c1c06b820 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_dump": [ 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.a", 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmcninMRAARhope6BpTuawcm/Bd/MSIg==", 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:56:23.377 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.b", 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmli3zARAAHVKo3x7gWq9M18P3MNqxXA==", 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.378 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.c", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmcqabOBAAqFr5SZkMULMCOz06c7V0qw==", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mds.d", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglm+6ITCRAACtqVZ3FpKoRXagsrFx3ywg==", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:56:23.379 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.0", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDVhglm8T+UJxAA3z+5rswh0d7RakAvfO7J8g==", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.1", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDYhglmke27LhAAF0oYalYC+cMjqc4Iw1r4kg==", 2024-03-31T15:56:23.380 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.2", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDbhglmcDWmNBAAuAJakIes6OBv3yzAhtT0DA==", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.3", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDehglm7Y42OxAA8+5GF4XF+3cTnxhi2BhiRw==", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.4", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDlhglmJwDhARAAzCpZr8/syvLjfnI0qi4jpA==", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.382 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.5", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDohglmGXmMBxAAQKoVfHNdxRWZIjtF3TN8CA==", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.6", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDrhglmjsnMDBAAYOGRa9UwgXIIjj6hbg8fwg==", 2024-03-31T15:56:23.383 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "osd.7", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDuhglmcWRFERAAWDwc/7Xlku2oXqBELqBCPg==", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow profile osd", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile osd", 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.0", 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A==", 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow r", 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.1", 2024-03-31T15:56:23.385 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmSpJzFhAAdwvdZGC6Tfafcb5kqX9/Zw==", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.2", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglmDJHvEhAA/EJD8+c4iK8u7ZEroAHp9g==", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.386 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.3", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDShglm2+uRHBAABJCOXHPmeC+WUichEcsMHw==", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow r", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow rw", 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow rwx" 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "client.admin", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmNbnzFRAAdnmwvGW9dCDLKNHHXy/8BA==", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": "allow *", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow *", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.x", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmdRVMLRAAHrLHygDnOeGTKMmMCVyzpg==", 2024-03-31T15:56:23.388 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "entity": "mgr.y", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "AQDRhglmy8KFKRAAyHCJHySxd3Jcb8lQBIA7eA==", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "caps": { 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "mds": "allow *", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "mon": "allow profile mgr", 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": "allow *" 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:23.389 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-03-31T15:56:23.390 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 msgr2=0x7f7bf806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 0x7f7bf806a840 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7c040af000 tx=0x7f7c040ab040 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 msgr2=0x7f7c1c15a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e980 tx=0x7f7c0c00ee40 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 shutdown_connections 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7bf8068380 0x7f7bf806a840 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c1c1449a0 0x7f7c1c0cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c1c0b3d90 0x7f7c1c163f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 --2- 172.21.15.90:0/962069439 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c1c06b820 0x7f7c1c15a880 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 >> 172.21.15.90:0/962069439 conn(0x7f7c1c0b1c80 msgr2=0x7f7c1c15d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:23.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 shutdown_connections 2024-03-31T15:56:23.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.384+0000 7f7c23800640 1 -- 172.21.15.90:0/962069439 wait complete. 2024-03-31T15:56:23.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 Processor -- start 2024-03-31T15:56:23.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 -- start start 2024-03-31T15:56:23.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f19d815b010 0x7f19d815f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19d80723c0 0x7f19d8072820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d815eb10 con 0x7f19d806cb90 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f19d815b3f0 con 0x7f19d815b010 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19df000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f19d80708b0 con 0x7f19d80723c0 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f19d815b010 0x7f19d815f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f19d815b010 0x7f19d815f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41704/0 (socket says 172.21.15.90:41704) 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 -- 172.21.15.90:0/1443384812 learned_addr learned my addr 172.21.15.90:0/1443384812 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 -- 172.21.15.90:0/1443384812 >> v2:172.21.15.102:3300/0 conn(0x7f19d80723c0 msgr2=0x7f19d8072820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 --2- 172.21.15.90:0/1443384812 >> v2:172.21.15.102:3300/0 conn(0x7f19d80723c0 0x7f19d8072820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 -- 172.21.15.90:0/1443384812 >> v1:172.21.15.102:6789/0 conn(0x7f19d806cb90 legacy=0x7f19d80cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.540+0000 7f19d7e00640 1 -- 172.21.15.90:0/1443384812 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d8072f30 con 0x7f19d815b010 2024-03-31T15:56:23.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19d7e00640 1 --2- 172.21.15.90:0/1443384812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d815b010 0x7f19d815f5b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f19d809f0d0 tx=0x7f19c801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9c2f19854173fc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19d7400640 1 -- 172.21.15.90:0/1443384812 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80b1070 con 0x7f19d815b010 2024-03-31T15:56:23.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19d7400640 1 -- 172.21.15.90:0/1443384812 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c80ac630 con 0x7f19d815b010 2024-03-31T15:56:23.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19d7400640 1 -- 172.21.15.90:0/1443384812 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80a8050 con 0x7f19d815b010 2024-03-31T15:56:23.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 -- 172.21.15.90:0/1443384812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d815b010 msgr2=0x7f19d815f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 --2- 172.21.15.90:0/1443384812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d815b010 0x7f19d815f5b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f19d809f0d0 tx=0x7f19c801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 -- 172.21.15.90:0/1443384812 shutdown_connections 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 --2- 172.21.15.90:0/1443384812 >> v2:172.21.15.102:3300/0 conn(0x7f19d80723c0 0x7f19d8072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 --2- 172.21.15.90:0/1443384812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d815b010 0x7f19d815f5b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 -- 172.21.15.90:0/1443384812 >> 172.21.15.90:0/1443384812 conn(0x7f19d80d13a0 msgr2=0x7f19d80d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 -- 172.21.15.90:0/1443384812 shutdown_connections 2024-03-31T15:56:23.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.544+0000 7f19df000640 1 -- 172.21.15.90:0/1443384812 wait complete. 2024-03-31T15:56:23.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 Processor -- start 2024-03-31T15:56:23.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- start start 2024-03-31T15:56:23.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 0x7f19d806edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19d815b010 0x7f19d806f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8165e00 con 0x7f19d806cb90 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f19d806d3f0 con 0x7f19d80723c0 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f19d8165f80 con 0x7f19d815b010 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41720/0 (socket says 172.21.15.90:41720) 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 -- 172.21.15.90:0/1199356096 learned_addr learned my addr 172.21.15.90:0/1199356096 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19cf400640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 0x7f19d806edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19dd000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19d815b010 0x7f19d806f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 -- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19d815b010 msgr2=0x7f19d806f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19d815b010 0x7f19d806f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 -- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 msgr2=0x7f19d806edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 0x7f19d806edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 -- 172.21.15.90:0/1199356096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d8167ce0 con 0x7f19d806cb90 2024-03-31T15:56:23.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19cf400640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 0x7f19d806edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d7e00640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f19c80af040 tx=0x7f19c8019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80bc070 con 0x7f19d806cb90 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c80b7430 con 0x7f19d806cb90 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80b1040 con 0x7f19d806cb90 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d8167f80 con 0x7f19d806cb90 2024-03-31T15:56:23.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d8151b20 con 0x7f19d806cb90 2024-03-31T15:56:23.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.548+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d80686d0 con 0x7f19d806cb90 2024-03-31T15:56:23.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.552+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f19c80bf070 con 0x7f19d806cb90 2024-03-31T15:56:23.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.552+0000 7f19d4c00640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 0x7f19b406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.552+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5249+0+0 (secure 0 0 0) 0x7f19c811ff50 con 0x7f19d806cb90 2024-03-31T15:56:23.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.552+0000 7f19cf400640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 0x7f19b406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.552+0000 7f19cf400640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 0x7f19b406a840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19d80b6970 tx=0x7f19c0009570 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.556+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f19c80160d0 con 0x7f19d806cb90 2024-03-31T15:56:23.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.712+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19d80cf6d0 con 0x7f19d806cb90 2024-03-31T15:56:23.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.712+0000 7f19d4c00640 1 -- 172.21.15.90:0/1199356096 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+13128 (secure 0 0 0) 0x7f19c80f0630 con 0x7f19d806cb90 2024-03-31T15:56:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:23.707 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":51,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:22.998970+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 msgr2=0x7f19b406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 0x7f19b406a840 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19d80b6970 tx=0x7f19c0009570 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 msgr2=0x7f19d81515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f19c80af040 tx=0x7f19c8019280 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 shutdown_connections 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f19b4068380 0x7f19b406a840 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f19d815b010 0x7f19d806f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d80723c0 0x7f19d806edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 --2- 172.21.15.90:0/1199356096 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19d806cb90 0x7f19d81515e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 >> 172.21.15.90:0/1199356096 conn(0x7f19d80d13a0 msgr2=0x7f19d806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:23.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 shutdown_connections 2024-03-31T15:56:23.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.716+0000 7f19df000640 1 -- 172.21.15.90:0/1199356096 wait complete. 2024-03-31T15:56:23.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:23.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:23.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 Processor -- start 2024-03-31T15:56:23.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 -- start start 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0b0071940 0x7fd0b015b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b0072a60 con 0x7fd0b015b040 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0b00aeac0 con 0x7fd0b00725a0 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0b4e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd0b00ae890 con 0x7fd0b0071940 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ade00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd0b015b040 0x7fd0b0071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50368/0 (socket says 172.21.15.90:50368) 2024-03-31T15:56:23.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ade00640 1 -- 172.21.15.90:0/561734926 learned_addr learned my addr 172.21.15.90:0/561734926 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824314918 0 0) 0x7fd0b0072a60 con 0x7fd0b015b040 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ae800640 1 --2- 172.21.15.90:0/561734926 >> v2:172.21.15.102:3300/0 conn(0x7fd0b0071940 0x7fd0b015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd094003530 con 0x7fd0b015b040 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3462935576 0 0) 0x7fd094003530 con 0x7fd0b015b040 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 >> v2:172.21.15.102:3300/0 conn(0x7fd0b0071940 msgr2=0x7fd0b015b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 --2- 172.21.15.90:0/561734926 >> v2:172.21.15.102:3300/0 conn(0x7fd0b0071940 0x7fd0b015b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 >> v1:172.21.15.102:6789/0 conn(0x7fd0b00725a0 legacy=0x7fd0b00b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b00aeda0 con 0x7fd0b015b040 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fd0a0001a00 con 0x7fd0b015b040 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 >> v1:172.21.15.90:6789/0 conn(0x7fd0b015b040 legacy=0x7fd0b0071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940039c0 0x7fd094003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd094004810 0x7fd094004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.872+0000 7fd0ad400640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 0x7fd09400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400c040 con 0x7fd0940051d0 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400c1f0 con 0x7fd094004810 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd09400c3f0 con 0x7fd0940039c0 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 0x7fd09400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ade00640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940039c0 0x7fd094003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0a5400640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd094004810 0x7fd094004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 -- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940039c0 msgr2=0x7fd094003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940039c0 0x7fd094003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 -- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd094004810 msgr2=0x7fd094004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd094004810 0x7fd094004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 -- 172.21.15.90:0/561734926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd094003530 con 0x7fd0940051d0 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 -- 172.21.15.90:0/561734926 reap_dead start 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ae800640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 0x7fd09400b9d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009ae0 tx=0x7fd0a801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fae2e6b6a1c2860 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80096b0 con 0x7fd0940051d0 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a8004020 con 0x7fd0940051d0 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.876+0000 7fd0ad400640 1 -- 172.21.15.90:0/561734926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8004410 con 0x7fd0940051d0 2024-03-31T15:56:23.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 msgr2=0x7fd09400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 0x7fd09400b9d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009ae0 tx=0x7fd0a801cab0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- 172.21.15.90:0/561734926 shutdown_connections 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/561734926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0940051d0 0x7fd09400b9d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- 172.21.15.90:0/561734926 >> 172.21.15.90:0/561734926 conn(0x7fd0b00b18b0 msgr2=0x7fd0b00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- 172.21.15.90:0/561734926 shutdown_connections 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- 172.21.15.90:0/561734926 wait complete. 2024-03-31T15:56:23.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 Processor -- start 2024-03-31T15:56:23.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- start start 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940051d0 0x7fd0b0143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0b01f24a0 0x7fd0b014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b00b35c0 con 0x7fd0b01f2920 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0072d40 con 0x7fd0b01f24a0 2024-03-31T15:56:23.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0b4e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd0b00713f0 con 0x7fd0940051d0 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41748/0 (socket says 172.21.15.90:41748) 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 -- 172.21.15.90:0/2628966638 learned_addr learned my addr 172.21.15.90:0/2628966638 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ade00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940051d0 0x7fd0b0143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0a5400640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0b01f24a0 0x7fd0b014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 -- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940051d0 msgr2=0x7fd0b0143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940051d0 0x7fd0b0143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 -- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0b01f24a0 msgr2=0x7fd0b014e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0b01f24a0 0x7fd0b014e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 -- 172.21.15.90:0/2628966638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0b01f1400 con 0x7fd0b01f2920 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.880+0000 7fd0ae800640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80984c0 tx=0x7fd094005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80b2070 con 0x7fd0b01f2920 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a80bb070 con 0x7fd0b01f2920 2024-03-31T15:56:23.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80096b0 con 0x7fd0b01f2920 2024-03-31T15:56:23.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0b01f16e0 con 0x7fd0b01f2920 2024-03-31T15:56:23.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0b01f1ca0 con 0x7fd0b01f2920 2024-03-31T15:56:23.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0b0075880 con 0x7fd0b01f2920 2024-03-31T15:56:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd0a80a5cf0 con 0x7fd0b01f2920 2024-03-31T15:56:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 0x7fd08c06a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5249+0+0 (secure 0 0 0) 0x7fd0a81204c0 con 0x7fd0b01f2920 2024-03-31T15:56:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0ade00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 0x7fd08c06a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:23.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.884+0000 7fd0ade00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 0x7fd08c06a810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0000f90 tx=0x7fd0a0006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:23.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:23.888+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd0a80c4020 con 0x7fd0b01f2920 2024-03-31T15:56:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.048+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd0b0157e60 con 0x7fd0b01f2920 2024-03-31T15:56:24.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.048+0000 7fd0a6a00640 1 -- 172.21.15.90:0/2628966638 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) ==== 93+0+10 (secure 0 0 0) 0x7fd0a80167c0 con 0x7fd0b01f2920 2024-03-31T15:56:24.043 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 msgr2=0x7fd08c06a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 0x7fd08c06a810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0000f90 tx=0x7fd0a0006040 comp rx=0 tx=0).stop 2024-03-31T15:56:24.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 msgr2=0x7fd0b014ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80984c0 tx=0x7fd094005840 comp rx=0 tx=0).stop 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 shutdown_connections 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd08c068350 0x7fd08c06a810 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd0b01f2920 0x7fd0b014ea50 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0b01f24a0 0x7fd0b014e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 --2- 172.21.15.90:0/2628966638 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd0940051d0 0x7fd0b0143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 >> 172.21.15.90:0/2628966638 conn(0x7fd0b00b18b0 msgr2=0x7fd0b00b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 shutdown_connections 2024-03-31T15:56:24.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.052+0000 7fd0b4e00640 1 -- 172.21.15.90:0/2628966638 wait complete. 2024-03-31T15:56:24.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-31T15:56:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.204+0000 7f620ca00640 1 Processor -- start 2024-03-31T15:56:24.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 -- start start 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f620815e300 0x7f6208161d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f620806d210 0x7f620806eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6208070a60 con 0x7f6208162280 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f620806e3e0 con 0x7f620806d210 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f620ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f62081628b0 con 0x7f620815e300 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6206800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f620806d210 0x7f620806eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f61ffe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6208162280 0x7f620806cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50378/0 (socket says 172.21.15.90:50378) 2024-03-31T15:56:24.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f61ffe00640 1 -- 172.21.15.90:0/4168479087 learned_addr learned my addr 172.21.15.90:0/4168479087 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6205e00640 1 --2- 172.21.15.90:0/4168479087 >> v2:172.21.15.102:3300/0 conn(0x7f620815e300 0x7f6208161d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6206800640 1 -- 172.21.15.90:0/4168479087 >> v2:172.21.15.102:3300/0 conn(0x7f620815e300 msgr2=0x7f6208161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6206800640 1 --2- 172.21.15.90:0/4168479087 >> v2:172.21.15.102:3300/0 conn(0x7f620815e300 0x7f6208161d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6206800640 1 -- 172.21.15.90:0/4168479087 >> v1:172.21.15.90:6789/0 conn(0x7f6208162280 legacy=0x7f620806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6205400640 1 -- 172.21.15.90:0/4168479087 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 434263924 0 0) 0x7f6208070a60 con 0x7f6208162280 2024-03-31T15:56:24.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.208+0000 7f6206800640 1 -- 172.21.15.90:0/4168479087 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6208162b90 con 0x7f620806d210 2024-03-31T15:56:24.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f6206800640 1 --2- 172.21.15.90:0/4168479087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620806d210 0x7f620806eea0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f62080ce8f0 tx=0x7f620001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b89239722db2e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f6205400640 1 -- 172.21.15.90:0/4168479087 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62000b2070 con 0x7f620806d210 2024-03-31T15:56:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f6205400640 1 -- 172.21.15.90:0/4168479087 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62000a57f0 con 0x7f620806d210 2024-03-31T15:56:24.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f6205400640 1 -- 172.21.15.90:0/4168479087 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62000a5bc0 con 0x7f620806d210 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- 172.21.15.90:0/4168479087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620806d210 msgr2=0x7f620806eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 --2- 172.21.15.90:0/4168479087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620806d210 0x7f620806eea0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f62080ce8f0 tx=0x7f620001c930 comp rx=0 tx=0).stop 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- 172.21.15.90:0/4168479087 shutdown_connections 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 --2- 172.21.15.90:0/4168479087 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620806d210 0x7f620806eea0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 --2- 172.21.15.90:0/4168479087 >> v2:172.21.15.102:3300/0 conn(0x7f620815e300 0x7f6208161d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- 172.21.15.90:0/4168479087 >> 172.21.15.90:0/4168479087 conn(0x7f62080d13a0 msgr2=0x7f62080d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- 172.21.15.90:0/4168479087 shutdown_connections 2024-03-31T15:56:24.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- 172.21.15.90:0/4168479087 wait complete. 2024-03-31T15:56:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 Processor -- start 2024-03-31T15:56:24.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.212+0000 7f620ca00640 1 -- start start 2024-03-31T15:56:24.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f620806d210 0x7f62081fae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6208162280 0x7f62080c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f620806ff80 con 0x7f620815e300 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f620806fe00 con 0x7f620806d210 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6208070100 con 0x7f6208162280 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41766/0 (socket says 172.21.15.90:41766) 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 -- 172.21.15.90:0/2757872386 learned_addr learned my addr 172.21.15.90:0/2757872386 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 -- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6208162280 msgr2=0x7f62080c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6208162280 0x7f62080c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 -- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f620806d210 msgr2=0x7f62081fae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f620806d210 0x7f62081fae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 -- 172.21.15.90:0/2757872386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62081575a0 con 0x7f620815e300 2024-03-31T15:56:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61ffe00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f61f00011f0 tx=0x7f61f000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0018070 con 0x7f620815e300 2024-03-31T15:56:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f00074e0 con 0x7f620815e300 2024-03-31T15:56:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0004a10 con 0x7f620815e300 2024-03-31T15:56:24.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6208144190 con 0x7f620815e300 2024-03-31T15:56:24.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6208144750 con 0x7f620815e300 2024-03-31T15:56:24.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6208155e50 con 0x7f620815e300 2024-03-31T15:56:24.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.216+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61f0004bb0 con 0x7f620815e300 2024-03-31T15:56:24.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.220+0000 7f61fe000640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 0x7f61e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.220+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5249+0+0 (secure 0 0 0) 0x7f61f0085c30 con 0x7f620815e300 2024-03-31T15:56:24.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.220+0000 7f6205e00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 0x7f61e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.220+0000 7f6205e00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 0x7f61e406a840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f62080b1c40 tx=0x7f61f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.224+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f61f0056310 con 0x7f620815e300 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f62080686d0 con 0x7f620815e300 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f61fe000640 1 -- 172.21.15.90:0/2757872386 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) ==== 84+0+65 (secure 0 0 0) 0x7f61f0055c10 con 0x7f620815e300 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stdout:[ 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "poolnum": 1, 2024-03-31T15:56:24.377 INFO:teuthology.orchestra.run.smithi090.stdout: "poolname": ".mgr" 2024-03-31T15:56:24.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:24.378 INFO:teuthology.orchestra.run.smithi090.stdout:] 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 msgr2=0x7f61e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 0x7f61e406a840 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f62080b1c40 tx=0x7f61f8008040 comp rx=0 tx=0).stop 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 msgr2=0x7f62080b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f61f00011f0 tx=0x7f61f000c9b0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 shutdown_connections 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61e4068380 0x7f61e406a840 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6208162280 0x7f62080c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f620815e300 0x7f62080b1170 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 --2- 172.21.15.90:0/2757872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f620806d210 0x7f62081fae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 >> 172.21.15.90:0/2757872386 conn(0x7f62080d13a0 msgr2=0x7f62080d0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 shutdown_connections 2024-03-31T15:56:24.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.384+0000 7f620ca00640 1 -- 172.21.15.90:0/2757872386 wait complete. 2024-03-31T15:56:24.398 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-31T15:56:24.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-31T15:56:24.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 Processor -- start 2024-03-31T15:56:24.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 -- start start 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 0x7f2a8c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8c0cb0e0 con 0x7f2a8c15e1f0 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8c0ceed0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a91200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8c0ce350 con 0x7f2a8c0ca380 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41768/0 (socket says 172.21.15.90:41768) 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 -- 172.21.15.90:0/1253901882 learned_addr learned my addr 172.21.15.90:0/1253901882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89e00640 1 --2- 172.21.15.90:0/1253901882 >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 0x7f2a8c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 -- 172.21.15.90:0/1253901882 >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 msgr2=0x7f2a8c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 --2- 172.21.15.90:0/1253901882 >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 0x7f2a8c0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.544+0000 7f2a89400640 1 -- 172.21.15.90:0/1253901882 >> v1:172.21.15.102:6789/0 conn(0x7f2a8c15e1f0 legacy=0x7f2a8c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:24.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a89400640 1 -- 172.21.15.90:0/1253901882 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c0ce630 con 0x7f2a8c0ca760 2024-03-31T15:56:24.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a89e00640 1 --2- 172.21.15.90:0/1253901882 >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 0x7f2a8c0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:24.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a89400640 1 --2- 172.21.15.90:0/1253901882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b2940 tx=0x7f2a7801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06b96508d086d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a88a00640 1 -- 172.21.15.90:0/1253901882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780a6040 con 0x7f2a8c0ca760 2024-03-31T15:56:24.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a88a00640 1 -- 172.21.15.90:0/1253901882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a780af070 con 0x7f2a8c0ca760 2024-03-31T15:56:24.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a88a00640 1 -- 172.21.15.90:0/1253901882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780aa860 con 0x7f2a8c0ca760 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 -- 172.21.15.90:0/1253901882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 msgr2=0x7f2a8c15a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 --2- 172.21.15.90:0/1253901882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0b2940 tx=0x7f2a7801c4e0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 -- 172.21.15.90:0/1253901882 shutdown_connections 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 --2- 172.21.15.90:0/1253901882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c15a9c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 --2- 172.21.15.90:0/1253901882 >> v2:172.21.15.102:3300/0 conn(0x7f2a8c0ca380 0x7f2a8c0cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 -- 172.21.15.90:0/1253901882 >> 172.21.15.90:0/1253901882 conn(0x7f2a8c0b1880 msgr2=0x7f2a8c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 -- 172.21.15.90:0/1253901882 shutdown_connections 2024-03-31T15:56:24.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.548+0000 7f2a91200640 1 -- 172.21.15.90:0/1253901882 wait complete. 2024-03-31T15:56:24.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 Processor -- start 2024-03-31T15:56:24.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- start start 2024-03-31T15:56:24.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2a8c0ca380 0x7f2a8c0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a8c15e1f0 0x7f2a8c14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c165580 con 0x7f2a8c0ca760 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c0cabc0 con 0x7f2a8c15e1f0 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8c165700 con 0x7f2a8c0ca380 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41780/0 (socket says 172.21.15.90:41780) 2024-03-31T15:56:24.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 -- 172.21.15.90:0/3380329938 learned_addr learned my addr 172.21.15.90:0/3380329938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:24.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 -- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2a8c0ca380 msgr2=0x7f2a8c0b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89e00640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2a8c0ca380 0x7f2a8c0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2a8c0ca380 0x7f2a8c0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 -- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a8c15e1f0 msgr2=0x7f2a8c14d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a8c15e1f0 0x7f2a8c14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 -- 172.21.15.90:0/3380329938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c14ef10 con 0x7f2a8c0ca760 2024-03-31T15:56:24.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a89400640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2a7801a110 tx=0x7f2a7801ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780bd5f0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a780a6030 con 0x7f2a8c0ca760 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8c14f1f0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780bd990 con 0x7f2a8c0ca760 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.552+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8c14f7b0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.556+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a8c0686d0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.556+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a780bdb30 con 0x7f2a8c0ca760 2024-03-31T15:56:24.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.556+0000 7f2a82000640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 0x7f2a6c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:24.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.556+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5249+0+0 (secure 0 0 0) 0x7f2a78120a40 con 0x7f2a8c0ca760 2024-03-31T15:56:24.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.556+0000 7f2a89e00640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 0x7f2a6c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:24.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.560+0000 7f2a89e00640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 0x7f2a6c06a840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c006fd0 tx=0x7f2a7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:24.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.560+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2a780160d0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:24.720+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2a8c0686d0 con 0x7f2a8c0ca760 2024-03-31T15:56:24.999 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.004+0000 7f2a82000640 1 -- 172.21.15.90:0/3380329938 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v52) ==== 138+0+0 (secure 0 0 0) 0x7f2a780bde20 con 0x7f2a8c0ca760 2024-03-31T15:56:24.999 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' created 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 msgr2=0x7f2a6c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 0x7f2a6c06a840 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c006fd0 tx=0x7f2a7c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 msgr2=0x7f2a8c14cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2a7801a110 tx=0x7f2a7801ad20 comp rx=0 tx=0).stop 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 shutdown_connections 2024-03-31T15:56:25.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2a6c068380 0x7f2a6c06a840 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2a8c15e1f0 0x7f2a8c14d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2a8c0ca760 0x7f2a8c14cf60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 --2- 172.21.15.90:0/3380329938 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2a8c0ca380 0x7f2a8c0b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 >> 172.21.15.90:0/3380329938 conn(0x7f2a8c0b1880 msgr2=0x7f2a8c0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 shutdown_connections 2024-03-31T15:56:25.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.008+0000 7f2a91200640 1 -- 172.21.15.90:0/3380329938 wait complete. 2024-03-31T15:56:25.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-31T15:56:25.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 Processor -- start 2024-03-31T15:56:25.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 -- start start 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd17015c290 0x7fd1700ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 0x7fd1700aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd17015fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd17006e9c0 con 0x7fd17015c290 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd17006de40 con 0x7fd1700ad3d0 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd177200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd17006e120 con 0x7fd1700af040 2024-03-31T15:56:25.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd17015c290 0x7fd1700ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd17015c290 0x7fd1700ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41788/0 (socket says 172.21.15.90:41788) 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 -- 172.21.15.90:0/2659961696 learned_addr learned my addr 172.21.15.90:0/2659961696 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd175200640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd17015fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16f400640 1 --2- 172.21.15.90:0/2659961696 >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 0x7fd1700aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 -- 172.21.15.90:0/2659961696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 msgr2=0x7fd17015fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd17015fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 -- 172.21.15.90:0/2659961696 >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 msgr2=0x7fd1700aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 --2- 172.21.15.90:0/2659961696 >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 0x7fd1700aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16fe00640 1 -- 172.21.15.90:0/2659961696 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17006e400 con 0x7fd17015c290 2024-03-31T15:56:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.176+0000 7fd16f400640 1 --2- 172.21.15.90:0/2659961696 >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 0x7fd1700aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:25.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd16fe00640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd17015c290 0x7fd1700ace90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd170070e00 tx=0x7fd16001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d16fd208c42233 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:25.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd16ea00640 1 -- 172.21.15.90:0/2659961696 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600b0070 con 0x7fd17015c290 2024-03-31T15:56:25.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd16ea00640 1 -- 172.21.15.90:0/2659961696 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1600ab600 con 0x7fd17015c290 2024-03-31T15:56:25.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd16ea00640 1 -- 172.21.15.90:0/2659961696 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600ab9d0 con 0x7fd17015c290 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 -- 172.21.15.90:0/2659961696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd17015c290 msgr2=0x7fd1700ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd17015c290 0x7fd1700ace90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd170070e00 tx=0x7fd16001c260 comp rx=0 tx=0).stop 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 -- 172.21.15.90:0/2659961696 shutdown_connections 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd17015fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 --2- 172.21.15.90:0/2659961696 >> v2:172.21.15.102:3300/0 conn(0x7fd1700ad3d0 0x7fd1700aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 --2- 172.21.15.90:0/2659961696 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd17015c290 0x7fd1700ace90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 -- 172.21.15.90:0/2659961696 >> 172.21.15.90:0/2659961696 conn(0x7fd1700b18b0 msgr2=0x7fd1700b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 -- 172.21.15.90:0/2659961696 shutdown_connections 2024-03-31T15:56:25.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.180+0000 7fd177200640 1 -- 172.21.15.90:0/2659961696 wait complete. 2024-03-31T15:56:25.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 Processor -- start 2024-03-31T15:56:25.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- start start 2024-03-31T15:56:25.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd170152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17015c290 0x7fd170157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd170071750 con 0x7fd1700ad3d0 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd17006dc00 con 0x7fd17015c290 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd1700718d0 con 0x7fd1700af040 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41802/0 (socket says 172.21.15.90:41802) 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 -- 172.21.15.90:0/1715614157 learned_addr learned my addr 172.21.15.90:0/1715614157 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16f400640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd170152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd175200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17015c290 0x7fd170157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 -- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 msgr2=0x7fd170152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd170152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 -- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17015c290 msgr2=0x7fd170157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17015c290 0x7fd170157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 -- 172.21.15.90:0/1715614157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1700b0df0 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd16fe00640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd16001a910 tx=0x7fd16001aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600ba020 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1600b6420 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1600b0040 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17015ae20 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd17015b3e0 con 0x7fd1700ad3d0 2024-03-31T15:56:25.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.184+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd170075880 con 0x7fd1700ad3d0 2024-03-31T15:56:25.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.188+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1600bd020 con 0x7fd1700ad3d0 2024-03-31T15:56:25.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.188+0000 7fd167e00640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 0x7fd15006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:25.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.188+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5619+0+0 (secure 0 0 0) 0x7fd160120d90 con 0x7fd1700ad3d0 2024-03-31T15:56:25.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.188+0000 7fd16f400640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 0x7fd15006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:25.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.188+0000 7fd16f400640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 0x7fd15006a840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd1700cea70 tx=0x7fd15c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:25.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.192+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd1600c4030 con 0x7fd1700ad3d0 2024-03-31T15:56:25.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:25.352+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd1700685f0 con 0x7fd1700ad3d0 2024-03-31T15:56:26.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.012+0000 7fd167e00640 1 -- 172.21.15.90:0/1715614157 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v53) ==== 170+0+0 (secure 0 0 0) 0x7fd160016110 con 0x7fd1700ad3d0 2024-03-31T15:56:26.006 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' created 2024-03-31T15:56:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 msgr2=0x7fd15006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 0x7fd15006a840 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd1700cea70 tx=0x7fd15c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:26.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 msgr2=0x7fd17015f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd16001a910 tx=0x7fd16001aaa0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 shutdown_connections 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd150068380 0x7fd15006a840 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17015c290 0x7fd170157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd1700af040 0x7fd170152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 --2- 172.21.15.90:0/1715614157 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1700ad3d0 0x7fd17015f0a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 >> 172.21.15.90:0/1715614157 conn(0x7fd1700b18b0 msgr2=0x7fd1701588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 shutdown_connections 2024-03-31T15:56:26.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.016+0000 7fd177200640 1 -- 172.21.15.90:0/1715614157 wait complete. 2024-03-31T15:56:26.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-31T15:56:26.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 Processor -- start 2024-03-31T15:56:26.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 -- start start 2024-03-31T15:56:26.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feeac15e690 0x7feeac0d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 0x7feeac06d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeac0d0390 con 0x7feeac06daf0 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7feeac0d0630 con 0x7feeac15e690 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeb1e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7feeac0d0910 con 0x7feeac15ea70 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feeac15e690 0x7feeac0d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7feeac15e690 0x7feeac0d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41818/0 (socket says 172.21.15.90:41818) 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeab400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7feeac06daf0 0x7feeac0ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50386/0 (socket says 172.21.15.90:50386) 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 -- 172.21.15.90:0/4037377810 learned_addr learned my addr 172.21.15.90:0/4037377810 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feea3e00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 0x7feeac06d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 -- 172.21.15.90:0/4037377810 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 msgr2=0x7feeac06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 0x7feeac06d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 -- 172.21.15.90:0/4037377810 >> v1:172.21.15.90:6789/0 conn(0x7feeac06daf0 legacy=0x7feeac0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaaa00640 1 -- 172.21.15.90:0/4037377810 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac0d0bf0 con 0x7feeac15e690 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feeaa000640 1 -- 172.21.15.90:0/4037377810 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1412683497 0 0) 0x7feeac0d0390 con 0x7feeac06daf0 2024-03-31T15:56:26.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.188+0000 7feea3e00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 0x7feeac06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeaaa00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15e690 0x7feeac0d16d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efaf9a7324cf463c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeaa000640 1 -- 172.21.15.90:0/4037377810 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0ab680 con 0x7feeac15e690 2024-03-31T15:56:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeaa000640 1 -- 172.21.15.90:0/4037377810 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee9c0b0070 con 0x7feeac15e690 2024-03-31T15:56:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeaa000640 1 -- 172.21.15.90:0/4037377810 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0aba40 con 0x7feeac15e690 2024-03-31T15:56:26.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 -- 172.21.15.90:0/4037377810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15e690 msgr2=0x7feeac0d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15e690 0x7feeac0d16d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee9c006fd0 tx=0x7fee9c01c480 comp rx=0 tx=0).stop 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 -- 172.21.15.90:0/4037377810 shutdown_connections 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac15ea70 0x7feeac06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 --2- 172.21.15.90:0/4037377810 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15e690 0x7feeac0d16d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 -- 172.21.15.90:0/4037377810 >> 172.21.15.90:0/4037377810 conn(0x7feeac0d1f20 msgr2=0x7feeac0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 -- 172.21.15.90:0/4037377810 shutdown_connections 2024-03-31T15:56:26.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.192+0000 7feeb1e00640 1 -- 172.21.15.90:0/4037377810 wait complete. 2024-03-31T15:56:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 Processor -- start 2024-03-31T15:56:26.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- start start 2024-03-31T15:56:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 0x7feeac0b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 0x7feeac14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feeac0cf030 con 0x7feeac15ea70 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feeac0ceeb0 con 0x7feeac15e690 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7feeac0ccd80 con 0x7feeac06daf0 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41822/0 (socket says 172.21.15.90:41822) 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 -- 172.21.15.90:0/1463105688 learned_addr learned my addr 172.21.15.90:0/1463105688 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feea3e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 0x7feeac14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeaaa00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 0x7feeac0b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 -- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 msgr2=0x7feeac0b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 0x7feeac0b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 -- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 msgr2=0x7feeac14f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 0x7feeac14f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 -- 172.21.15.90:0/1463105688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac14f9f0 con 0x7feeac15ea70 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feea3e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 0x7feeac14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:26.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeaaa00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 0x7feeac0b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeab400640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7feea400c7a0 tx=0x7feea400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4018070 con 0x7feeac15ea70 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea400a640 con 0x7feeac15ea70 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea400aa70 con 0x7feeac15ea70 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac14ff90 con 0x7feeac15ea70 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac15c9d0 con 0x7feeac15ea70 2024-03-31T15:56:26.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.196+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeac15cbd0 con 0x7feeac15ea70 2024-03-31T15:56:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.200+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feea401b070 con 0x7feeac15ea70 2024-03-31T15:56:26.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.200+0000 7feea3400640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 0x7fee8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.200+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 6001+0+0 (secure 0 0 0) 0x7feea40863b0 con 0x7feeac15ea70 2024-03-31T15:56:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.200+0000 7feeaaa00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 0x7fee8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:26.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.200+0000 7feeaaa00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 0x7fee8806a840 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0af000 tx=0x7fee9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:26.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.204+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7feeac15cbd0 con 0x7feeac15ea70 2024-03-31T15:56:26.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:26.392+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7feeac0685f0 con 0x7feeac15ea70 2024-03-31T15:56:27.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.012+0000 7feea3400640 1 -- 172.21.15.90:0/1463105688 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v23) ==== 172+0+0 (secure 0 0 0) 0x7feea40568b0 con 0x7feeac15ea70 2024-03-31T15:56:27.008 INFO:teuthology.orchestra.run.smithi090.stderr:new fs with metadata pool 8 and data pool 9 2024-03-31T15:56:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 msgr2=0x7fee8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 0x7fee8806a840 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0af000 tx=0x7fee9c009040 comp rx=0 tx=0).stop 2024-03-31T15:56:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 msgr2=0x7feeac155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7feea400c7a0 tx=0x7feea400cc60 comp rx=0 tx=0).stop 2024-03-31T15:56:27.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 shutdown_connections 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee88068380 0x7fee8806a840 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feeac15ea70 0x7feeac155240 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feeac15e690 0x7feeac14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 --2- 172.21.15.90:0/1463105688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7feeac06daf0 0x7feeac0b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 >> 172.21.15.90:0/1463105688 conn(0x7feeac0d1f20 msgr2=0x7feeac06f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 shutdown_connections 2024-03-31T15:56:27.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.016+0000 7feeb1e00640 1 -- 172.21.15.90:0/1463105688 wait complete. 2024-03-31T15:56:27.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:27.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.196+0000 7fc641600640 1 Processor -- start 2024-03-31T15:56:27.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.196+0000 7fc641600640 1 -- start start 2024-03-31T15:56:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc641600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc63c0cc9d0 0x7fc63c142030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc641600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc641600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63c0b34f0 con 0x7fc63c0ccdb0 2024-03-31T15:56:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc641600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc63c0b3ab0 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc641600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c141ba0 con 0x7fc63c06b820 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc63c0cc9d0 0x7fc63c142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:38074/0 (socket says 172.21.15.90:38074) 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 -- 172.21.15.90:0/2845313191 learned_addr learned my addr 172.21.15.90:0/2845313191 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc632000640 1 --1- 172.21.15.90:0/2845313191 >> v1:172.21.15.102:6789/0 conn(0x7fc63c0ccdb0 0x7fc63c0b2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:53566/0 (socket says 172.21.15.90:53566) 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 -- 172.21.15.90:0/2845313191 >> v2:172.21.15.102:3300/0 conn(0x7fc63c0cc9d0 msgr2=0x7fc63c142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 --2- 172.21.15.90:0/2845313191 >> v2:172.21.15.102:3300/0 conn(0x7fc63c0cc9d0 0x7fc63c142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 -- 172.21.15.90:0/2845313191 >> v1:172.21.15.102:6789/0 conn(0x7fc63c0ccdb0 legacy=0x7fc63c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.200+0000 7fc63b400640 1 -- 172.21.15.90:0/2845313191 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0b37d0 con 0x7fc63c06b820 2024-03-31T15:56:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc63b400640 1 --2- 172.21.15.90:0/2845313191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc634009e50 tx=0x7fc63401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe2786d831719aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc63a000640 1 -- 172.21.15.90:0/2845313191 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6340a5660 con 0x7fc63c06b820 2024-03-31T15:56:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc63a000640 1 -- 172.21.15.90:0/2845313191 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6340a5800 con 0x7fc63c06b820 2024-03-31T15:56:27.197 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc63a000640 1 -- 172.21.15.90:0/2845313191 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6340a5bd0 con 0x7fc63c06b820 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- 172.21.15.90:0/2845313191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 msgr2=0x7fc63c06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 --2- 172.21.15.90:0/2845313191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc634009e50 tx=0x7fc63401c9a0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- 172.21.15.90:0/2845313191 shutdown_connections 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 --2- 172.21.15.90:0/2845313191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c06bcb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 --2- 172.21.15.90:0/2845313191 >> v2:172.21.15.102:3300/0 conn(0x7fc63c0cc9d0 0x7fc63c142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- 172.21.15.90:0/2845313191 >> 172.21.15.90:0/2845313191 conn(0x7fc63c0b1c80 msgr2=0x7fc63c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- 172.21.15.90:0/2845313191 shutdown_connections 2024-03-31T15:56:27.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- 172.21.15.90:0/2845313191 wait complete. 2024-03-31T15:56:27.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 Processor -- start 2024-03-31T15:56:27.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.204+0000 7fc641600640 1 -- start start 2024-03-31T15:56:27.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c13d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ccdb0 0x7fc63c1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0cf080 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0cef00 con 0x7fc63c0ccdb0 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c15c620 con 0x7fc63c06b820 2024-03-31T15:56:27.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41838/0 (socket says 172.21.15.90:41838) 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 -- 172.21.15.90:0/2452632191 learned_addr learned my addr 172.21.15.90:0/2452632191 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc63b400640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ccdb0 0x7fc63c1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc63aa00640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c13d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 -- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 msgr2=0x7fc63c13d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c13d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 -- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ccdb0 msgr2=0x7fc63c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ccdb0 0x7fc63c1408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 -- 172.21.15.90:0/2452632191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c13c550 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc632000640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c370 tx=0x7fc62400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6240074e0 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc624007680 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c13c830 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624004a30 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c13cdf0 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c0686d0 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.208+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc62401a020 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.212+0000 7fc633400640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 0x7fc61806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.212+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fc6240863d0 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.212+0000 7fc63aa00640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 0x7fc61806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.212+0000 7fc63aa00640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 0x7fc61806a840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009af0 tx=0x7fc62c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.216+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc624056760 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc63c152b10 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc633400640 1 -- 172.21.15.90:0/2452632191 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+16915 (secure 0 0 0) 0x7fc624056060 con 0x7fc63c0cc9d0 2024-03-31T15:56:27.370 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:27.370 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":54,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:26.396356+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 msgr2=0x7fc61806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 0x7fc61806a840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009af0 tx=0x7fc62c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 msgr2=0x7fc63c140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc62400c370 tx=0x7fc62400c830 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 shutdown_connections 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc618068380 0x7fc61806a840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc63c0ccdb0 0x7fc63c1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc63c0cc9d0 0x7fc63c140390 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 --2- 172.21.15.90:0/2452632191 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc63c06b820 0x7fc63c13d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 >> 172.21.15.90:0/2452632191 conn(0x7fc63c0b1c80 msgr2=0x7fc63c0af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 shutdown_connections 2024-03-31T15:56:27.372 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.376+0000 7fc641600640 1 -- 172.21.15.90:0/2452632191 wait complete. 2024-03-31T15:56:27.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:27.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.532+0000 7f4c85200640 1 Processor -- start 2024-03-31T15:56:27.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.532+0000 7f4c85200640 1 -- start start 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c85200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 0x7f4c800d0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c85200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c8006b5f0 0x7f4c8015fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c85200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c8006bd90 con 0x7f4c801600f0 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c85200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f4c80160560 con 0x7f4c8006b5f0 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c85200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4c8006c9a0 con 0x7f4c801424a0 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c8006b5f0 0x7f4c8015fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f4c8006b5f0 0x7f4c8015fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41842/0 (socket says 172.21.15.90:41842) 2024-03-31T15:56:27.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 -- 172.21.15.90:0/1185286869 learned_addr learned my addr 172.21.15.90:0/1185286869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7de00640 1 --2- 172.21.15.90:0/1185286869 >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 0x7f4c800d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 -- 172.21.15.90:0/1185286869 >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 msgr2=0x7f4c800d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 --2- 172.21.15.90:0/1185286869 >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 0x7f4c800d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 -- 172.21.15.90:0/1185286869 >> v1:172.21.15.90:6789/0 conn(0x7f4c801600f0 legacy=0x7f4c8015c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 -- 172.21.15.90:0/1185286869 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c8006c070 con 0x7f4c8006b5f0 2024-03-31T15:56:27.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7d400640 1 -- 172.21.15.90:0/1185286869 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236862840 0 0) 0x7f4c8006bd90 con 0x7f4c801600f0 2024-03-31T15:56:27.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7de00640 1 --2- 172.21.15.90:0/1185286869 >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 0x7f4c800d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:27.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c75400640 1 --2- 172.21.15.90:0/1185286869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c8006b5f0 0x7f4c8015fbb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c800b58f0 tx=0x7f4c6801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5cbbef2af9f0772 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7d400640 1 -- 172.21.15.90:0/1185286869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680b2070 con 0x7f4c8006b5f0 2024-03-31T15:56:27.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7d400640 1 -- 172.21.15.90:0/1185286869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c680ad680 con 0x7f4c8006b5f0 2024-03-31T15:56:27.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.536+0000 7f4c7d400640 1 -- 172.21.15.90:0/1185286869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680ada50 con 0x7f4c8006b5f0 2024-03-31T15:56:27.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- 172.21.15.90:0/1185286869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c8006b5f0 msgr2=0x7f4c8015fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- 172.21.15.90:0/1185286869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c8006b5f0 0x7f4c8015fbb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4c800b58f0 tx=0x7f4c6801c490 comp rx=0 tx=0).stop 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- 172.21.15.90:0/1185286869 shutdown_connections 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- 172.21.15.90:0/1185286869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c8006b5f0 0x7f4c8015fbb0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- 172.21.15.90:0/1185286869 >> v2:172.21.15.102:3300/0 conn(0x7f4c801424a0 0x7f4c800d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- 172.21.15.90:0/1185286869 >> 172.21.15.90:0/1185286869 conn(0x7f4c800d0650 msgr2=0x7f4c800d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- 172.21.15.90:0/1185286869 shutdown_connections 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- 172.21.15.90:0/1185286869 wait complete. 2024-03-31T15:56:27.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 Processor -- start 2024-03-31T15:56:27.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- start start 2024-03-31T15:56:27.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 0x7f4c80152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 0x7f4c8013aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8006cf90 con 0x7f4c80159640 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8006ce10 con 0x7f4c8006b5f0 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c85200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f4c80168f50 con 0x7f4c801600f0 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41850/0 (socket says 172.21.15.90:41850) 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 -- 172.21.15.90:0/18638141 learned_addr learned my addr 172.21.15.90:0/18638141 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7de00640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 0x7f4c80152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c75400640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 0x7f4c8013aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 -- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 msgr2=0x7f4c8013aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 0x7f4c8013aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 -- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 msgr2=0x7f4c80152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 0x7f4c80152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c7e800640 1 -- 172.21.15.90:0/18638141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c80159190 con 0x7f4c80159640 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.540+0000 7f4c75400640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 0x7f4c8013aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c7de00640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 0x7f4c80152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c7e800640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c780 tx=0x7f4c7800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78003ba0 con 0x7f4c80159640 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c78003d60 con 0x7f4c80159640 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800aa90 con 0x7f4c80159640 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c80157920 con 0x7f4c80159640 2024-03-31T15:56:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c80153150 con 0x7f4c80159640 2024-03-31T15:56:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c8013b7b0 con 0x7f4c80159640 2024-03-31T15:56:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4c7800ac30 con 0x7f4c80159640 2024-03-31T15:56:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 0x7f4c5c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7f4c780869d0 con 0x7f4c80159640 2024-03-31T15:56:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.544+0000 7f4c7de00640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 0x7f4c5c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.548+0000 7f4c7de00640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 0x7f4c5c06a870 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000b990 tx=0x7f4c70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.548+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f4c78056e90 con 0x7f4c80159640 2024-03-31T15:56:27.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.708+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c800686d0 con 0x7f4c80159640 2024-03-31T15:56:27.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.708+0000 7f4c76a00640 1 -- 172.21.15.90:0/18638141 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+16915 (secure 0 0 0) 0x7f4c78056790 con 0x7f4c80159640 2024-03-31T15:56:27.703 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:27.704 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":54,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:26.396356+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:27.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 msgr2=0x7f4c5c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 0x7f4c5c06a870 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000b990 tx=0x7f4c70008040 comp rx=0 tx=0).stop 2024-03-31T15:56:27.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 msgr2=0x7f4c8013af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c780 tx=0x7f4c7800cc40 comp rx=0 tx=0).stop 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 shutdown_connections 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f4c5c0683b0 0x7f4c5c06a870 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4c80159640 0x7f4c8013af70 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f4c801600f0 0x7f4c8013aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 --2- 172.21.15.90:0/18638141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c8006b5f0 0x7f4c80152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 >> 172.21.15.90:0/18638141 conn(0x7f4c800d0650 msgr2=0x7f4c8015a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 shutdown_connections 2024-03-31T15:56:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.712+0000 7f4c85200640 1 -- 172.21.15.90:0/18638141 wait complete. 2024-03-31T15:56:27.725 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-31T15:56:27.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.868+0000 7fcf04000640 1 Processor -- start 2024-03-31T15:56:27.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.868+0000 7fcf04000640 1 -- start start 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf04000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf04000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcefc0acf50 0x7fcefc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf04000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcefc06ed20 con 0x7fcefc0b4210 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf04000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcefc0cc110 con 0x7fcefc0acf50 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf04000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcefc0cc3f0 con 0x7fcefc15a710 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf02000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcefc0b4210 0x7fcefc15e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50408/0 (socket says 172.21.15.90:50408) 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf02000640 1 -- 172.21.15.90:0/1564550705 learned_addr learned my addr 172.21.15.90:0/1564550705 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 --2- 172.21.15.90:0/1564550705 >> v2:172.21.15.102:3300/0 conn(0x7fcefc0acf50 0x7fcefc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf01600640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549071439 0 0) 0x7fcefc06ed20 con 0x7fcefc0b4210 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee4003530 con 0x7fcefc0b4210 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 -- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 msgr2=0x7fcefc06d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:27.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc06d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 -- 172.21.15.90:0/1564550705 >> v1:172.21.15.90:6789/0 conn(0x7fcefc0b4210 legacy=0x7fcefc15e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1803132625 0 0) 0x7fcee4003530 con 0x7fcefc0b4210 2024-03-31T15:56:27.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 -- 172.21.15.90:0/1564550705 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc0b2bb0 con 0x7fcefc0acf50 2024-03-31T15:56:27.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf01600640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc06d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.872+0000 7fcf00c00640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0acf50 0x7fcefc06ddb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcee800a960 tx=0x7fcee801d870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a01fa25a7942dba server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b3070 con 0x7fcefc0acf50 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee80a6710 con 0x7fcefc0acf50 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcefbe00640 1 -- 172.21.15.90:0/1564550705 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee800b040 con 0x7fcefc0acf50 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0acf50 msgr2=0x7fcefc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0acf50 0x7fcefc06ddb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcee800a960 tx=0x7fcee801d870 comp rx=0 tx=0).stop 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- 172.21.15.90:0/1564550705 shutdown_connections 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0acf50 0x7fcefc06ddb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- 172.21.15.90:0/1564550705 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc06d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- 172.21.15.90:0/1564550705 >> 172.21.15.90:0/1564550705 conn(0x7fcefc0b18b0 msgr2=0x7fcefc0b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- 172.21.15.90:0/1564550705 shutdown_connections 2024-03-31T15:56:27.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- 172.21.15.90:0/1564550705 wait complete. 2024-03-31T15:56:27.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 Processor -- start 2024-03-31T15:56:27.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- start start 2024-03-31T15:56:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 0x7fcefc15b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc0ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc15eba0 con 0x7fcefc0acf50 2024-03-31T15:56:27.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc15ea20 con 0x7fcefc0b4210 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf04000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcefc15ed20 con 0x7fcefc15a710 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf01600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf01600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41856/0 (socket says 172.21.15.90:41856) 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf01600640 1 -- 172.21.15.90:0/17951783 learned_addr learned my addr 172.21.15.90:0/17951783 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.876+0000 7fcf00c00640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 0x7fcefc15b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf02000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc0ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 -- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 msgr2=0x7fcefc0ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc0ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 -- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 msgr2=0x7fcefc15b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 0x7fcefc15b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 -- 172.21.15.90:0/17951783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc163560 con 0x7fcefc0acf50 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf00c00640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 0x7fcefc15b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf02000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf01600640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e980 tx=0x7fcef000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0017070 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef000ca00 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef000cdd0 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcefc150fb0 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcefc163ae0 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcefc075880 con 0x7fcefc0acf50 2024-03-31T15:56:27.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcef001a070 con 0x7fcefc0acf50 2024-03-31T15:56:27.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.880+0000 7fcef9600640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 0x7fcedc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:27.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.884+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6101+0+0 (secure 0 0 0) 0x7fcef0086aa0 con 0x7fcefc0acf50 2024-03-31T15:56:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.884+0000 7fcf00c00640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 0x7fcedc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:27.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.884+0000 7fcf00c00640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 0x7fcedc06a840 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcee80b0000 tx=0x7fcee800c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:27.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:27.888+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fcef0056ec0 con 0x7fcefc0acf50 2024-03-31T15:56:28.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:28.080+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fcefc06e550 con 0x7fcefc0acf50 2024-03-31T15:56:28.987 INFO:tasks.ceph.mds.a.smithi090.stderr:2024-03-31T15:56:28.992+0000 7f1df4800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:56:29.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcef9600640 1 -- 172.21.15.90:0/17951783 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7fcef00567c0 con 0x7fcefc0acf50 2024-03-31T15:56:29.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 msgr2=0x7fcedc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 0x7fcedc06a840 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcee80b0000 tx=0x7fcee800c040 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 msgr2=0x7fcefc150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcef000e980 tx=0x7fcef000ee40 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 shutdown_connections 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcedc068380 0x7fcedc06a840 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcefc15a710 0x7fcefc0ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcefc0b4210 0x7fcefc15b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 --2- 172.21.15.90:0/17951783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcefc0acf50 0x7fcefc150a70 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 >> 172.21.15.90:0/17951783 conn(0x7fcefc0b18b0 msgr2=0x7fcefc06d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 shutdown_connections 2024-03-31T15:56:29.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.048+0000 7fcf04000640 1 -- 172.21.15.90:0/17951783 wait complete. 2024-03-31T15:56:29.063 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-31T15:56:29.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-31T15:56:29.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 Processor -- start 2024-03-31T15:56:29.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 -- start start 2024-03-31T15:56:29.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0c980d14c0 0x7f0c980d1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:29.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9806c350 con 0x7f0c9815e780 2024-03-31T15:56:29.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c98073ab0 con 0x7f0c9806cf00 2024-03-31T15:56:29.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.208+0000 7f0c9f800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9806dc80 con 0x7f0c980d14c0 2024-03-31T15:56:29.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c9ce00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0c9815e780 0x7f0c980cc5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50412/0 (socket says 172.21.15.90:50412) 2024-03-31T15:56:29.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c9ce00640 1 -- 172.21.15.90:0/3428072611 learned_addr learned my addr 172.21.15.90:0/3428072611 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97e00640 1 --2- 172.21.15.90:0/3428072611 >> v2:172.21.15.102:3300/0 conn(0x7f0c980d14c0 0x7f0c980d1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97400640 1 -- 172.21.15.90:0/3428072611 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 253042557 0 0) 0x7f0c9806c350 con 0x7f0c9815e780 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97400640 1 -- 172.21.15.90:0/3428072611 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c80003530 con 0x7f0c9815e780 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97e00640 1 -- 172.21.15.90:0/3428072611 >> v1:172.21.15.102:6789/0 conn(0x7f0c9806cf00 legacy=0x7f0c9806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97e00640 1 -- 172.21.15.90:0/3428072611 >> v1:172.21.15.90:6789/0 conn(0x7f0c9815e780 legacy=0x7f0c980cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97e00640 1 -- 172.21.15.90:0/3428072611 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9806cae0 con 0x7f0c980d14c0 2024-03-31T15:56:29.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97e00640 1 --2- 172.21.15.90:0/3428072611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c980d14c0 0x7f0c980d1910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097c0 tx=0x7f0c8401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b3a0eb1144f7e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:29.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.212+0000 7f0c97400640 1 -- 172.21.15.90:0/3428072611 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840b0070 con 0x7f0c980d14c0 2024-03-31T15:56:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c97400640 1 -- 172.21.15.90:0/3428072611 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c840ab600 con 0x7f0c980d14c0 2024-03-31T15:56:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c97400640 1 -- 172.21.15.90:0/3428072611 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840ab9d0 con 0x7f0c980d14c0 2024-03-31T15:56:29.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- 172.21.15.90:0/3428072611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c980d14c0 msgr2=0x7f0c980d1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 --2- 172.21.15.90:0/3428072611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c980d14c0 0x7f0c980d1910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097c0 tx=0x7f0c8401c410 comp rx=0 tx=0).stop 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- 172.21.15.90:0/3428072611 shutdown_connections 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 --2- 172.21.15.90:0/3428072611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c980d14c0 0x7f0c980d1910 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- 172.21.15.90:0/3428072611 >> 172.21.15.90:0/3428072611 conn(0x7f0c980cea00 msgr2=0x7f0c980cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- 172.21.15.90:0/3428072611 shutdown_connections 2024-03-31T15:56:29.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- 172.21.15.90:0/3428072611 wait complete. 2024-03-31T15:56:29.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 Processor -- start 2024-03-31T15:56:29.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- start start 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 0x7f0c98158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0c9815e780 0x7f0c980b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0c980ccce0 con 0x7f0c980d14c0 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0c980cc290 con 0x7f0c9806cf00 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9f800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0c98156f50 con 0x7f0c9815e780 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41862/0 (socket says 172.21.15.90:41862) 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c97e00640 1 -- 172.21.15.90:0/877341507 learned_addr learned my addr 172.21.15.90:0/877341507 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:29.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9ce00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 0x7f0c98158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.216+0000 7f0c9d800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0c9815e780 0x7f0c980b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 -- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0c9815e780 msgr2=0x7f0c980b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0c9815e780 0x7f0c980b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 -- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 msgr2=0x7f0c98158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 0x7f0c98158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 -- 172.21.15.90:0/877341507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c98158920 con 0x7f0c980d14c0 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c9ce00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 0x7f0c98158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:29.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c97e00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0c84019020 tx=0x7f0c840191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840ba070 con 0x7f0c980d14c0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c840b43f0 con 0x7f0c980d14c0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c98158c00 con 0x7f0c980d14c0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840b0040 con 0x7f0c980d14c0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c981591c0 con 0x7f0c980d14c0 2024-03-31T15:56:29.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c980685f0 con 0x7f0c980d14c0 2024-03-31T15:56:29.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.220+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0c840bd070 con 0x7f0c980d14c0 2024-03-31T15:56:29.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.224+0000 7f0c94c00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 0x7f0c7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:29.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.224+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0c8411fa40 con 0x7f0c980d14c0 2024-03-31T15:56:29.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.224+0000 7f0c9ce00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 0x7f0c7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:29.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.224+0000 7f0c9ce00640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 0x7f0c7806a840 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0010f0 tx=0x7f0c8c007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:29.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.228+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0c84016090 con 0x7f0c980d14c0 2024-03-31T15:56:29.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:29.424+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0c9806e7a0 con 0x7f0c980d14c0 2024-03-31T15:56:30.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.052+0000 7f0c94c00640 1 -- 172.21.15.90:0/877341507 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) ==== 122+0+0 (secure 0 0 0) 0x7f0c8411fe20 con 0x7f0c980d14c0 2024-03-31T15:56:30.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 msgr2=0x7f0c7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 0x7f0c7806a840 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0010f0 tx=0x7f0c8c007bf0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 msgr2=0x7f0c98156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0c84019020 tx=0x7f0c840191b0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 shutdown_connections 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0c78068380 0x7f0c7806a840 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0c9815e780 0x7f0c980b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0c980d14c0 0x7f0c98156a10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 --2- 172.21.15.90:0/877341507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9806cf00 0x7f0c98158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 >> 172.21.15.90:0/877341507 conn(0x7f0c980cea00 msgr2=0x7f0c980cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 shutdown_connections 2024-03-31T15:56:30.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.056+0000 7f0c9f800640 1 -- 172.21.15.90:0/877341507 wait complete. 2024-03-31T15:56:30.076 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-31T15:56:30.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:30.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 Processor -- start 2024-03-31T15:56:30.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 -- start start 2024-03-31T15:56:30.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 0x7f25f815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.213 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25f80cca30 0x7f25f80ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25f8160350 con 0x7f25f80d2680 2024-03-31T15:56:30.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f25f8160bf0 con 0x7f25f80cca30 2024-03-31T15:56:30.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25fcc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25f8160630 con 0x7f25f815c120 2024-03-31T15:56:30.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 0x7f25f815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25f80cca30 0x7f25f80ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25efe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f25f80d2680 0x7f25f80b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:53582/0 (socket says 172.21.15.90:53582) 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25efe00640 1 -- 172.21.15.90:0/1581231576 learned_addr learned my addr 172.21.15.90:0/1581231576 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5e00640 1 -- 172.21.15.90:0/1581231576 >> v2:172.21.15.102:3300/0 conn(0x7f25f80cca30 msgr2=0x7f25f80ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5e00640 1 --2- 172.21.15.90:0/1581231576 >> v2:172.21.15.102:3300/0 conn(0x7f25f80cca30 0x7f25f80ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5e00640 1 -- 172.21.15.90:0/1581231576 >> v1:172.21.15.102:6789/0 conn(0x7f25f80d2680 legacy=0x7f25f80b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5400640 1 -- 172.21.15.90:0/1581231576 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1151712882 0 0) 0x7f25f8160350 con 0x7f25f80d2680 2024-03-31T15:56:30.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.220+0000 7f25f5e00640 1 -- 172.21.15.90:0/1581231576 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f8160910 con 0x7f25f815c120 2024-03-31T15:56:30.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25f5e00640 1 --2- 172.21.15.90:0/1581231576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 0x7f25f815fd00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097c0 tx=0x7f25e801ce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb4e8db3573ba44d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25f5400640 1 -- 172.21.15.90:0/1581231576 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e80b0070 con 0x7f25f815c120 2024-03-31T15:56:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25f5400640 1 -- 172.21.15.90:0/1581231576 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e80ab600 con 0x7f25f815c120 2024-03-31T15:56:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25f5400640 1 -- 172.21.15.90:0/1581231576 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e80ab9d0 con 0x7f25f815c120 2024-03-31T15:56:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 -- 172.21.15.90:0/1581231576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 msgr2=0x7f25f815fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.218 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 --2- 172.21.15.90:0/1581231576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 0x7f25f815fd00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f25e80097c0 tx=0x7f25e801ce70 comp rx=0 tx=0).stop 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 -- 172.21.15.90:0/1581231576 shutdown_connections 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 --2- 172.21.15.90:0/1581231576 >> v2:172.21.15.102:3300/0 conn(0x7f25f80cca30 0x7f25f80ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 --2- 172.21.15.90:0/1581231576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f815c120 0x7f25f815fd00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 -- 172.21.15.90:0/1581231576 >> 172.21.15.90:0/1581231576 conn(0x7f25f809aa40 msgr2=0x7f25f809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 -- 172.21.15.90:0/1581231576 shutdown_connections 2024-03-31T15:56:30.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.224+0000 7f25fcc00640 1 -- 172.21.15.90:0/1581231576 wait complete. 2024-03-31T15:56:30.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 Processor -- start 2024-03-31T15:56:30.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- start start 2024-03-31T15:56:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f80cca30 0x7f25f80c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25f815c120 0x7f25f80c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25f80b2f70 con 0x7f25f80d2680 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25f80b2df0 con 0x7f25f815c120 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25f8169040 con 0x7f25f80cca30 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41874/0 (socket says 172.21.15.90:41874) 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 -- 172.21.15.90:0/2701120130 learned_addr learned my addr 172.21.15.90:0/2701120130 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25f5e00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f80cca30 0x7f25f80c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25f6800640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25f815c120 0x7f25f80c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 -- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f80cca30 msgr2=0x7f25f80c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f80cca30 0x7f25f80c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 -- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25f815c120 msgr2=0x7f25f80c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25f815c120 0x7f25f80c7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 -- 172.21.15.90:0/2701120130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f80c8ff0 con 0x7f25f80d2680 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25efe00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c480 tx=0x7f25e000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0018070 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e0007500 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0004a30 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f80c9290 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f80cb7d0 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.228+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f80cb300 con 0x7f25f80d2680 2024-03-31T15:56:30.224 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.232+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25e0004bd0 con 0x7f25f80d2680 2024-03-31T15:56:30.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.232+0000 7f25ee000640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 0x7f25d406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.232+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f25e0085cd0 con 0x7f25f80d2680 2024-03-31T15:56:30.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.232+0000 7f25f5e00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 0x7f25d406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.232+0000 7f25f5e00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 0x7f25d406a870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f25e80ad000 tx=0x7f25e8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.236+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f25e0055aa0 con 0x7f25f80d2680 2024-03-31T15:56:30.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.424+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25f80686d0 con 0x7f25f80d2680 2024-03-31T15:56:30.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.424+0000 7f25ee000640 1 -- 172.21.15.90:0/2701120130 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7f25e0059da0 con 0x7f25f80d2680 2024-03-31T15:56:30.419 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:30.419 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:30.419 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 msgr2=0x7f25d406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 0x7f25d406a870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f25e80ad000 tx=0x7f25e8004740 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 msgr2=0x7f25f80c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c480 tx=0x7f25e000c940 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 shutdown_connections 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25d40683b0 0x7f25d406a870 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25f815c120 0x7f25f80c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25f80d2680 0x7f25f80c9ee0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 --2- 172.21.15.90:0/2701120130 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25f80cca30 0x7f25f80c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 >> 172.21.15.90:0/2701120130 conn(0x7f25f809aa40 msgr2=0x7f25f809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 shutdown_connections 2024-03-31T15:56:30.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.428+0000 7f25fcc00640 1 -- 172.21.15.90:0/2701120130 wait complete. 2024-03-31T15:56:30.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 Processor -- start 2024-03-31T15:56:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 -- start start 2024-03-31T15:56:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f018c0b4d90 0x7f018c1690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c1556b0 con 0x7f018c1552d0 2024-03-31T15:56:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c0ae4a0 con 0x7f018c070260 2024-03-31T15:56:30.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0193000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f018c0b41f0 con 0x7f018c0b4d90 2024-03-31T15:56:30.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f018c1552d0 0x7f018c06b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50422/0 (socket says 172.21.15.90:50422) 2024-03-31T15:56:30.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018be00640 1 -- 172.21.15.90:0/1524740806 learned_addr learned my addr 172.21.15.90:0/1524740806 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f0191000640 1 --2- 172.21.15.90:0/1524740806 >> v2:172.21.15.102:3300/0 conn(0x7f018c0b4d90 0x7f018c1690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898096474 0 0) 0x7f017c001120 con 0x7f018c1552d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0170003530 con 0x7f018c1552d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2661399579 0 0) 0x7f0170003530 con 0x7f018c1552d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 >> v2:172.21.15.102:3300/0 conn(0x7f018c0b4d90 msgr2=0x7f018c1690f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 --2- 172.21.15.90:0/1524740806 >> v2:172.21.15.102:3300/0 conn(0x7f018c0b4d90 0x7f018c1690f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 >> v1:172.21.15.102:6789/0 conn(0x7f018c070260 legacy=0x7f018c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c0ae780 con 0x7f018c1552d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f017c001850 con 0x7f018c1552d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 >> v1:172.21.15.90:6789/0 conn(0x7f018c1552d0 legacy=0x7f018c06b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01700039c0 0x7f0170003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0170004810 0x7f0170004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 0x7f017000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0b41f0 con 0x7f01700051d0 2024-03-31T15:56:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f017000c020 con 0x7f01700039c0 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.584+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f017000c220 con 0x7f0170004810 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 0x7f017000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f018be00640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01700039c0 0x7f0170003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f018b400640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0170004810 0x7f0170004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 -- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0170004810 msgr2=0x7f0170004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0170004810 0x7f0170004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 -- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01700039c0 msgr2=0x7f0170003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01700039c0 0x7f0170003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 -- 172.21.15.90:0/1524740806 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0170003530 con 0x7f01700051d0 2024-03-31T15:56:30.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 -- 172.21.15.90:0/1524740806 reap_dead start 2024-03-31T15:56:30.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f0191000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 0x7f017000b9d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0184009fc0 tx=0x7f018401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26f1980ee2e2d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184009c10 con 0x7f01700051d0 2024-03-31T15:56:30.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01840a5c00 con 0x7f01700051d0 2024-03-31T15:56:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.588+0000 7f018aa00640 1 -- 172.21.15.90:0/1524740806 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01840039a0 con 0x7f01700051d0 2024-03-31T15:56:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 msgr2=0x7f017000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 0x7f017000b9d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0184009fc0 tx=0x7f018401ccc0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- 172.21.15.90:0/1524740806 shutdown_connections 2024-03-31T15:56:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 --2- 172.21.15.90:0/1524740806 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01700051d0 0x7f017000b9d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- 172.21.15.90:0/1524740806 >> 172.21.15.90:0/1524740806 conn(0x7f018c0d0650 msgr2=0x7f018c0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- 172.21.15.90:0/1524740806 shutdown_connections 2024-03-31T15:56:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- 172.21.15.90:0/1524740806 wait complete. 2024-03-31T15:56:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 Processor -- start 2024-03-31T15:56:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- start start 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f01700051d0 0x7f018c14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018c151430 0x7f018c15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0ae2c0 con 0x7f018c06f270 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0b5240 con 0x7f018c151430 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0193000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f018c0b5ab0 con 0x7f01700051d0 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41896/0 (socket says 172.21.15.90:41896) 2024-03-31T15:56:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 -- 172.21.15.90:0/2632684665 learned_addr learned my addr 172.21.15.90:0/2632684665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f018b400640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018c151430 0x7f018c15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f018be00640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f01700051d0 0x7f018c14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 -- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f01700051d0 msgr2=0x7f018c14c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f01700051d0 0x7f018c14c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 -- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018c151430 msgr2=0x7f018c15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018c151430 0x7f018c15ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.592+0000 7f0191000640 1 -- 172.21.15.90:0/2632684665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c0674a0 con 0x7f018c06f270 2024-03-31T15:56:30.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0191000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f01840986e0 tx=0x7f0184009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184009c10 con 0x7f018c06f270 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01840bd070 con 0x7f018c06f270 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01840039a0 con 0x7f018c06f270 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c15b4b0 con 0x7f018c06f270 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c15ba70 con 0x7f018c06f270 2024-03-31T15:56:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f018c15bc40 con 0x7f018c06f270 2024-03-31T15:56:30.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.596+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0184003b40 con 0x7f018c06f270 2024-03-31T15:56:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.600+0000 7f0183e00640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 0x7f016c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.600+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0184122480 con 0x7f018c06f270 2024-03-31T15:56:30.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.600+0000 7f018be00640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 0x7f016c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.600+0000 7f018be00640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 0x7f016c06a840 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f017c0012c0 tx=0x7f017c007a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.604+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f01840169d0 con 0x7f018c06f270 2024-03-31T15:56:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f018c0686d0 con 0x7f018c06f270 2024-03-31T15:56:30.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0183e00640 1 -- 172.21.15.90:0/2632684665 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f01840f28b0 con 0x7f018c06f270 2024-03-31T15:56:30.758 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:30.758 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 msgr2=0x7f016c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 0x7f016c06a840 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f017c0012c0 tx=0x7f017c007a00 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 msgr2=0x7f018c06f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f01840986e0 tx=0x7f0184009750 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 shutdown_connections 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f016c068380 0x7f016c06a840 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f018c06f270 0x7f018c06f700 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018c151430 0x7f018c15ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 --2- 172.21.15.90:0/2632684665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f01700051d0 0x7f018c14c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 >> 172.21.15.90:0/2632684665 conn(0x7f018c0d0650 msgr2=0x7f018c0b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 shutdown_connections 2024-03-31T15:56:30.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.764+0000 7f0193000640 1 -- 172.21.15.90:0/2632684665 wait complete. 2024-03-31T15:56:30.779 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-31T15:56:30.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:56:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.920+0000 7f87cb800640 1 Processor -- start 2024-03-31T15:56:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.920+0000 7f87cb800640 1 -- start start 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.920+0000 7f87cb800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87c415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87cb800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87c4160350 con 0x7f87c40d2680 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87cb800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87c4160630 con 0x7f87c40cca30 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87cb800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f87c4160bf0 con 0x7f87c415c120 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f87c40d2680 0x7f87c40b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50430/0 (socket says 172.21.15.90:50430) 2024-03-31T15:56:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3e00640 1 -- 172.21.15.90:0/3172550726 learned_addr learned my addr 172.21.15.90:0/3172550726 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c8e00640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87c415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974413971 0 0) 0x7f87c4160350 con 0x7f87c40d2680 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87ac003530 con 0x7f87c40d2680 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522739361 0 0) 0x7f87c4160630 con 0x7f87c40cca30 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87c4160350 con 0x7f87c40cca30 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1676019836 0 0) 0x7f87ac003530 con 0x7f87c40d2680 2024-03-31T15:56:30.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 msgr2=0x7f87c415fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87c415fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 >> v1:172.21.15.102:6789/0 conn(0x7f87c40cca30 legacy=0x7f87c40d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c4160910 con 0x7f87c40d2680 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f87b0001680 con 0x7f87c40d2680 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 >> v1:172.21.15.90:6789/0 conn(0x7f87c40d2680 legacy=0x7f87c40b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87ac003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004130 0x7f87ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87ac004d50 0x7f87ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4160bf0 con 0x7f87ac004d50 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac00bb90 con 0x7f87ac004130 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac00bd90 con 0x7f87c415c120 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87ac004d50 0x7f87ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c8e00640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87ac003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c3e00640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004130 0x7f87ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 -- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 msgr2=0x7f87ac003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415c120 0x7f87ac003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 -- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004130 msgr2=0x7f87ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004130 0x7f87ac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.924+0000 7f87c9800640 1 -- 172.21.15.90:0/3172550726 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac003530 con 0x7f87ac004d50 2024-03-31T15:56:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87c9800640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87ac004d50 0x7f87ac0051d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001fd0 tx=0x7f87bc01f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f91f55490bc2e14 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc001c20 con 0x7f87ac004d50 2024-03-31T15:56:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc0a8cb0 con 0x7f87ac004d50 2024-03-31T15:56:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87c3400640 1 -- 172.21.15.90:0/3172550726 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0a8e60 con 0x7f87ac004d50 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- 172.21.15.90:0/3172550726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87ac004d50 msgr2=0x7f87ac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 --2- 172.21.15.90:0/3172550726 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87ac004d50 0x7f87ac0051d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001fd0 tx=0x7f87bc01f070 comp rx=0 tx=0).stop 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87c9800640 1 -- 172.21.15.90:0/3172550726 reap_dead start 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- 172.21.15.90:0/3172550726 shutdown_connections 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- 172.21.15.90:0/3172550726 >> 172.21.15.90:0/3172550726 conn(0x7f87c409aa40 msgr2=0x7f87c409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- 172.21.15.90:0/3172550726 shutdown_connections 2024-03-31T15:56:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- 172.21.15.90:0/3172550726 wait complete. 2024-03-31T15:56:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 Processor -- start 2024-03-31T15:56:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.928+0000 7f87cb800640 1 -- start start 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004d50 0x7f87c40b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415a940 0x7f87c415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f87c415c500 con 0x7f87c4158740 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87c40d2ae0 con 0x7f87ac004d50 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f87c415c680 con 0x7f87c415a940 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41918/0 (socket says 172.21.15.90:41918) 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 -- 172.21.15.90:0/451141921 learned_addr learned my addr 172.21.15.90:0/451141921 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c8e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004d50 0x7f87c40b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c9800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415a940 0x7f87c415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 -- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415a940 msgr2=0x7f87c415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415a940 0x7f87c415adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 -- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004d50 msgr2=0x7f87c40b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004d50 0x7f87c40b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 -- 172.21.15.90:0/451141921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c40c6400 con 0x7f87c4158740 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c3e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f87b0004e70 tx=0x7f87b0007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b001a070 con 0x7f87c4158740 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b000f750 con 0x7f87c4158740 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b000fb20 con 0x7f87c4158740 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c4157ad0 con 0x7f87c4158740 2024-03-31T15:56:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c4158f90 con 0x7f87c4158740 2024-03-31T15:56:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c40686d0 con 0x7f87c4158740 2024-03-31T15:56:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.932+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87b000fcc0 con 0x7f87c4158740 2024-03-31T15:56:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.936+0000 7f87c0c00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 0x7f87a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:30.928 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.936+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f87b00860e0 con 0x7f87c4158740 2024-03-31T15:56:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.936+0000 7f87c8e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 0x7f87a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:30.929 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.936+0000 7f87c8e00640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 0x7f87a406a840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009be0 tx=0x7f87b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:30.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:30.940+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f87b0055f50 con 0x7f87c4158740 2024-03-31T15:56:31.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.096+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87c415b300 con 0x7f87a4068380 2024-03-31T15:56:31.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.104+0000 7f87c0c00640 1 -- 172.21.15.90:0/451141921 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+243630 (secure 0 0 0) 0x7f87b8014c10 con 0x7f87a4068380 2024-03-31T15:56:31.100 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:31.101 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 msgr2=0x7f87a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 0x7f87a406a840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f87b8009be0 tx=0x7f87b8009190 comp rx=0 tx=0).stop 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 msgr2=0x7f87c4157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f87b0004e70 tx=0x7f87b0007430 comp rx=0 tx=0).stop 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 shutdown_connections 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87a4068380 0x7f87a406a840 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f87c415a940 0x7f87c415adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87c4158740 0x7f87c4157590 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 --2- 172.21.15.90:0/451141921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac004d50 0x7f87c40b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 >> 172.21.15.90:0/451141921 conn(0x7f87c409aa40 msgr2=0x7f87c40b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:31.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 shutdown_connections 2024-03-31T15:56:31.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.108+0000 7f87cb800640 1 -- 172.21.15.90:0/451141921 wait complete. 2024-03-31T15:56:31.120 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-03-31T15:56:29.254589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":126,"acting":126,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222156,"kb_used_data":5972,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752564,"statfs":{"total":773094113280,"available":772866625536,"internally_reserved":0,"allocated":6115328,"data_stored":3379321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.243852"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021551+0000","last_change":"2024-03-31T15:56:26.021551+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021551+0000","last_undegraded":"2024-03-31T15:56:26.021551+0000","last_fullsized":"2024-03-31T15:56:26.021551+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025926+0000","last_change":"2024-03-31T15:56:26.025926+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025926+0000","last_undegraded":"2024-03-31T15:56:26.025926+0000","last_fullsized":"2024-03-31T15:56:26.025926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028615+0000","last_change":"2024-03-31T15:56:26.028615+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028615+0000","last_undegraded":"2024-03-31T15:56:26.028615+0000","last_fullsized":"2024-03-31T15:56:26.028615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026552+0000","last_change":"2024-03-31T15:56:26.026552+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026552+0000","last_undegraded":"2024-03-31T15:56:26.026552+0000","last_fullsized":"2024-03-31T15:56:26.026552+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022133+0000","last_change":"2024-03-31T15:56:26.022133+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022133+0000","last_undegraded":"2024-03-31T15:56:26.022133+0000","last_fullsized":"2024-03-31T15:56:26.022133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022163+0000","last_change":"2024-03-31T15:56:26.022163+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022163+0000","last_undegraded":"2024-03-31T15:56:26.022163+0000","last_fullsized":"2024-03-31T15:56:26.022163+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021075+0000","last_change":"2024-03-31T15:56:26.021075+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021075+0000","last_undegraded":"2024-03-31T15:56:26.021075+0000","last_fullsized":"2024-03-31T15:56:26.021075+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028321+0000","last_change":"2024-03-31T15:56:26.028321+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028321+0000","last_undegraded":"2024-03-31T15:56:26.028321+0000","last_fullsized":"2024-03-31T15:56:26.028321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023949+0000","last_change":"2024-03-31T15:56:26.023949+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023949+0000","last_undegraded":"2024-03-31T15:56:26.023949+0000","last_fullsized":"2024-03-31T15:56:26.023949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027463+0000","last_change":"2024-03-31T15:56:26.027463+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027463+0000","last_undegraded":"2024-03-31T15:56:26.027463+0000","last_fullsized":"2024-03-31T15:56:26.027463+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025451+0000","last_change":"2024-03-31T15:56:26.025451+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025451+0000","last_undegraded":"2024-03-31T15:56:26.025451+0000","last_fullsized":"2024-03-31T15:56:26.025451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026315+0000","last_change":"2024-03-31T15:56:26.026315+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026315+0000","last_undegraded":"2024-03-31T15:56:26.026315+0000","last_fullsized":"2024-03-31T15:56:26.026315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026205+0000","last_change":"2024-03-31T15:56:26.026205+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026205+0000","last_undegraded":"2024-03-31T15:56:26.026205+0000","last_fullsized":"2024-03-31T15:56:26.026205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031242+0000","last_change":"2024-03-31T15:56:26.031242+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031242+0000","last_undegraded":"2024-03-31T15:56:26.031242+0000","last_fullsized":"2024-03-31T15:56:26.031242+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024954+0000","last_change":"2024-03-31T15:56:26.024954+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024954+0000","last_undegraded":"2024-03-31T15:56:26.024954+0000","last_fullsized":"2024-03-31T15:56:26.024954+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031116+0000","last_change":"2024-03-31T15:56:26.031116+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031116+0000","last_undegraded":"2024-03-31T15:56:26.031116+0000","last_fullsized":"2024-03-31T15:56:26.031116+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044934+0000","last_change":"2024-03-31T15:56:26.043587+0000","last_active":"2024-03-31T15:56:26.044934+0000","last_peered":"2024-03-31T15:56:26.044934+0000","last_clean":"2024-03-31T15:56:26.044934+0000","last_became_active":"2024-03-31T15:56:26.043321+0000","last_became_peered":"2024-03-31T15:56:26.043321+0000","last_unstale":"2024-03-31T15:56:26.044934+0000","last_undegraded":"2024-03-31T15:56:26.044934+0000","last_fullsized":"2024-03-31T15:56:26.044934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027162+0000","last_change":"2024-03-31T15:56:26.027162+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027162+0000","last_undegraded":"2024-03-31T15:56:26.027162+0000","last_fullsized":"2024-03-31T15:56:26.027162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.017667+0000","last_change":"2024-03-31T15:56:25.017667+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.017667+0000","last_undegraded":"2024-03-31T15:56:25.017667+0000","last_fullsized":"2024-03-31T15:56:25.017667+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026382+0000","last_change":"2024-03-31T15:56:26.026382+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026382+0000","last_undegraded":"2024-03-31T15:56:26.026382+0000","last_fullsized":"2024-03-31T15:56:26.026382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045804+0000","last_change":"2024-03-31T15:56:26.044595+0000","last_active":"2024-03-31T15:56:26.045804+0000","last_peered":"2024-03-31T15:56:26.045804+0000","last_clean":"2024-03-31T15:56:26.045804+0000","last_became_active":"2024-03-31T15:56:26.044321+0000","last_became_peered":"2024-03-31T15:56:26.044321+0000","last_unstale":"2024-03-31T15:56:26.045804+0000","last_undegraded":"2024-03-31T15:56:26.045804+0000","last_fullsized":"2024-03-31T15:56:26.045804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027580+0000","last_change":"2024-03-31T15:56:26.027580+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027580+0000","last_undegraded":"2024-03-31T15:56:26.027580+0000","last_fullsized":"2024-03-31T15:56:26.027580+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.038619+0000","last_change":"2024-03-31T15:56:26.035102+0000","last_active":"2024-03-31T15:56:26.038619+0000","last_peered":"2024-03-31T15:56:26.038619+0000","last_clean":"2024-03-31T15:56:26.038619+0000","last_became_active":"2024-03-31T15:56:26.034821+0000","last_became_peered":"2024-03-31T15:56:26.034821+0000","last_unstale":"2024-03-31T15:56:26.038619+0000","last_undegraded":"2024-03-31T15:56:26.038619+0000","last_fullsized":"2024-03-31T15:56:26.038619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042800+0000","last_change":"2024-03-31T15:56:26.035499+0000","last_active":"2024-03-31T15:56:26.042800+0000","last_peered":"2024-03-31T15:56:26.042800+0000","last_clean":"2024-03-31T15:56:26.042800+0000","last_became_active":"2024-03-31T15:56:26.035216+0000","last_became_peered":"2024-03-31T15:56:26.035216+0000","last_unstale":"2024-03-31T15:56:26.042800+0000","last_undegraded":"2024-03-31T15:56:26.042800+0000","last_fullsized":"2024-03-31T15:56:26.042800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044103+0000","last_change":"2024-03-31T15:56:26.040614+0000","last_active":"2024-03-31T15:56:26.044103+0000","last_peered":"2024-03-31T15:56:26.044103+0000","last_clean":"2024-03-31T15:56:26.044103+0000","last_became_active":"2024-03-31T15:56:26.040371+0000","last_became_peered":"2024-03-31T15:56:26.040371+0000","last_unstale":"2024-03-31T15:56:26.044103+0000","last_undegraded":"2024-03-31T15:56:26.044103+0000","last_fullsized":"2024-03-31T15:56:26.044103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025656+0000","last_change":"2024-03-31T15:56:26.025656+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025656+0000","last_undegraded":"2024-03-31T15:56:26.025656+0000","last_fullsized":"2024-03-31T15:56:26.025656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041171+0000","last_change":"2024-03-31T15:56:26.037304+0000","last_active":"2024-03-31T15:56:26.041171+0000","last_peered":"2024-03-31T15:56:26.041171+0000","last_clean":"2024-03-31T15:56:26.041171+0000","last_became_active":"2024-03-31T15:56:26.037038+0000","last_became_peered":"2024-03-31T15:56:26.037038+0000","last_unstale":"2024-03-31T15:56:26.041171+0000","last_undegraded":"2024-03-31T15:56:26.041171+0000","last_fullsized":"2024-03-31T15:56:26.041171+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:52.264596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045526+0000","last_change":"2024-03-31T15:56:26.040038+0000","last_active":"2024-03-31T15:56:26.045526+0000","last_peered":"2024-03-31T15:56:26.045526+0000","last_clean":"2024-03-31T15:56:26.045526+0000","last_became_active":"2024-03-31T15:56:26.039654+0000","last_became_peered":"2024-03-31T15:56:26.039654+0000","last_unstale":"2024-03-31T15:56:26.045526+0000","last_undegraded":"2024-03-31T15:56:26.045526+0000","last_fullsized":"2024-03-31T15:56:26.045526+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023782+0000","last_change":"2024-03-31T15:56:26.023782+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023782+0000","last_undegraded":"2024-03-31T15:56:26.023782+0000","last_fullsized":"2024-03-31T15:56:26.023782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:25.006583+0000","last_change":"2024-03-31T15:56:25.006583+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.006583+0000","last_undegraded":"2024-03-31T15:56:25.006583+0000","last_fullsized":"2024-03-31T15:56:25.006583+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.016843+0000","last_change":"2024-03-31T15:56:25.016843+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.016843+0000","last_undegraded":"2024-03-31T15:56:25.016843+0000","last_fullsized":"2024-03-31T15:56:25.016843+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023933+0000","last_change":"2024-03-31T15:56:26.023933+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023933+0000","last_undegraded":"2024-03-31T15:56:26.023933+0000","last_fullsized":"2024-03-31T15:56:26.023933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:25.006583+0000","last_change":"2024-03-31T15:56:25.006583+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.006583+0000","last_undegraded":"2024-03-31T15:56:25.006583+0000","last_fullsized":"2024-03-31T15:56:25.006583+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045554+0000","last_change":"2024-03-31T15:56:26.041173+0000","last_active":"2024-03-31T15:56:26.045554+0000","last_peered":"2024-03-31T15:56:26.045554+0000","last_clean":"2024-03-31T15:56:26.045554+0000","last_became_active":"2024-03-31T15:56:26.040901+0000","last_became_peered":"2024-03-31T15:56:26.040901+0000","last_unstale":"2024-03-31T15:56:26.045554+0000","last_undegraded":"2024-03-31T15:56:26.045554+0000","last_fullsized":"2024-03-31T15:56:26.045554+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022785+0000","last_change":"2024-03-31T15:56:26.022785+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022785+0000","last_undegraded":"2024-03-31T15:56:26.022785+0000","last_fullsized":"2024-03-31T15:56:26.022785+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.018463+0000","last_change":"2024-03-31T15:56:25.018463+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.018463+0000","last_undegraded":"2024-03-31T15:56:25.018463+0000","last_fullsized":"2024-03-31T15:56:25.018463+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048116+0000","last_change":"2024-03-31T15:56:26.041375+0000","last_active":"2024-03-31T15:56:26.048116+0000","last_peered":"2024-03-31T15:56:26.048116+0000","last_clean":"2024-03-31T15:56:26.048116+0000","last_became_active":"2024-03-31T15:56:26.041158+0000","last_became_peered":"2024-03-31T15:56:26.041158+0000","last_unstale":"2024-03-31T15:56:26.048116+0000","last_undegraded":"2024-03-31T15:56:26.048116+0000","last_fullsized":"2024-03-31T15:56:26.048116+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024021+0000","last_change":"2024-03-31T15:56:26.024021+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024021+0000","last_undegraded":"2024-03-31T15:56:26.024021+0000","last_fullsized":"2024-03-31T15:56:26.024021+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041150+0000","last_change":"2024-03-31T15:56:26.037434+0000","last_active":"2024-03-31T15:56:26.041150+0000","last_peered":"2024-03-31T15:56:26.041150+0000","last_clean":"2024-03-31T15:56:26.041150+0000","last_became_active":"2024-03-31T15:56:26.036753+0000","last_became_peered":"2024-03-31T15:56:26.036753+0000","last_unstale":"2024-03-31T15:56:26.041150+0000","last_undegraded":"2024-03-31T15:56:26.041150+0000","last_fullsized":"2024-03-31T15:56:26.041150+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.472784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:25.006583+0000","last_change":"2024-03-31T15:56:25.006583+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.006583+0000","last_undegraded":"2024-03-31T15:56:25.006583+0000","last_fullsized":"2024-03-31T15:56:25.006583+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.018416+0000","last_change":"2024-03-31T15:56:25.018416+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.018416+0000","last_undegraded":"2024-03-31T15:56:25.018416+0000","last_fullsized":"2024-03-31T15:56:25.018416+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027856+0000","last_change":"2024-03-31T15:56:26.027856+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027856+0000","last_undegraded":"2024-03-31T15:56:26.027856+0000","last_fullsized":"2024-03-31T15:56:26.027856+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042183+0000","last_change":"2024-03-31T15:56:26.039370+0000","last_active":"2024-03-31T15:56:26.042183+0000","last_peered":"2024-03-31T15:56:26.042183+0000","last_clean":"2024-03-31T15:56:26.042183+0000","last_became_active":"2024-03-31T15:56:26.038769+0000","last_became_peered":"2024-03-31T15:56:26.038769+0000","last_unstale":"2024-03-31T15:56:26.042183+0000","last_undegraded":"2024-03-31T15:56:26.042183+0000","last_fullsized":"2024-03-31T15:56:26.042183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:36.882522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041058+0000","last_change":"2024-03-31T15:56:26.036191+0000","last_active":"2024-03-31T15:56:26.041058+0000","last_peered":"2024-03-31T15:56:26.041058+0000","last_clean":"2024-03-31T15:56:26.041058+0000","last_became_active":"2024-03-31T15:56:26.035873+0000","last_became_peered":"2024-03-31T15:56:26.035873+0000","last_unstale":"2024-03-31T15:56:26.041058+0000","last_undegraded":"2024-03-31T15:56:26.041058+0000","last_fullsized":"2024-03-31T15:56:26.041058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045774+0000","last_change":"2024-03-31T15:56:26.044352+0000","last_active":"2024-03-31T15:56:26.045774+0000","last_peered":"2024-03-31T15:56:26.045774+0000","last_clean":"2024-03-31T15:56:26.045774+0000","last_became_active":"2024-03-31T15:56:26.044071+0000","last_became_peered":"2024-03-31T15:56:26.044071+0000","last_unstale":"2024-03-31T15:56:26.045774+0000","last_undegraded":"2024-03-31T15:56:26.045774+0000","last_fullsized":"2024-03-31T15:56:26.045774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022357+0000","last_change":"2024-03-31T15:56:26.022357+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022357+0000","last_undegraded":"2024-03-31T15:56:26.022357+0000","last_fullsized":"2024-03-31T15:56:26.022357+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.016939+0000","last_change":"2024-03-31T15:56:25.016939+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.016939+0000","last_undegraded":"2024-03-31T15:56:25.016939+0000","last_fullsized":"2024-03-31T15:56:25.016939+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025805+0000","last_change":"2024-03-31T15:56:26.025805+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025805+0000","last_undegraded":"2024-03-31T15:56:26.025805+0000","last_fullsized":"2024-03-31T15:56:26.025805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041491+0000","last_change":"2024-03-31T15:56:26.037979+0000","last_active":"2024-03-31T15:56:26.041491+0000","last_peered":"2024-03-31T15:56:26.041491+0000","last_clean":"2024-03-31T15:56:26.041491+0000","last_became_active":"2024-03-31T15:56:26.037707+0000","last_became_peered":"2024-03-31T15:56:26.037707+0000","last_unstale":"2024-03-31T15:56:26.041491+0000","last_undegraded":"2024-03-31T15:56:26.041491+0000","last_fullsized":"2024-03-31T15:56:26.041491+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026702+0000","last_change":"2024-03-31T15:56:26.026702+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026702+0000","last_undegraded":"2024-03-31T15:56:26.026702+0000","last_fullsized":"2024-03-31T15:56:26.026702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.036608+0000","last_change":"2024-03-31T15:56:26.030797+0000","last_active":"2024-03-31T15:56:26.036608+0000","last_peered":"2024-03-31T15:56:26.036608+0000","last_clean":"2024-03-31T15:56:26.036608+0000","last_became_active":"2024-03-31T15:56:26.030470+0000","last_became_peered":"2024-03-31T15:56:26.030470+0000","last_unstale":"2024-03-31T15:56:26.036608+0000","last_undegraded":"2024-03-31T15:56:26.036608+0000","last_fullsized":"2024-03-31T15:56:26.036608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025774+0000","last_change":"2024-03-31T15:56:26.025774+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025774+0000","last_undegraded":"2024-03-31T15:56:26.025774+0000","last_fullsized":"2024-03-31T15:56:26.025774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041438+0000","last_change":"2024-03-31T15:56:26.036552+0000","last_active":"2024-03-31T15:56:26.041438+0000","last_peered":"2024-03-31T15:56:26.041438+0000","last_clean":"2024-03-31T15:56:26.041438+0000","last_became_active":"2024-03-31T15:56:26.036319+0000","last_became_peered":"2024-03-31T15:56:26.036319+0000","last_unstale":"2024-03-31T15:56:26.041438+0000","last_undegraded":"2024-03-31T15:56:26.041438+0000","last_fullsized":"2024-03-31T15:56:26.041438+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021890+0000","last_change":"2024-03-31T15:56:26.021890+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021890+0000","last_undegraded":"2024-03-31T15:56:26.021890+0000","last_fullsized":"2024-03-31T15:56:26.021890+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042115+0000","last_change":"2024-03-31T15:56:26.039110+0000","last_active":"2024-03-31T15:56:26.042115+0000","last_peered":"2024-03-31T15:56:26.042115+0000","last_clean":"2024-03-31T15:56:26.042115+0000","last_became_active":"2024-03-31T15:56:26.037645+0000","last_became_peered":"2024-03-31T15:56:26.037645+0000","last_unstale":"2024-03-31T15:56:26.042115+0000","last_undegraded":"2024-03-31T15:56:26.042115+0000","last_fullsized":"2024-03-31T15:56:26.042115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:37:11.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023537+0000","last_change":"2024-03-31T15:56:26.023537+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023537+0000","last_undegraded":"2024-03-31T15:56:26.023537+0000","last_fullsized":"2024-03-31T15:56:26.023537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048112+0000","last_change":"2024-03-31T15:56:26.040713+0000","last_active":"2024-03-31T15:56:26.048112+0000","last_peered":"2024-03-31T15:56:26.048112+0000","last_clean":"2024-03-31T15:56:26.048112+0000","last_became_active":"2024-03-31T15:56:26.040391+0000","last_became_peered":"2024-03-31T15:56:26.040391+0000","last_unstale":"2024-03-31T15:56:26.048112+0000","last_undegraded":"2024-03-31T15:56:26.048112+0000","last_fullsized":"2024-03-31T15:56:26.048112+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027172+0000","last_change":"2024-03-31T15:56:26.027172+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027172+0000","last_undegraded":"2024-03-31T15:56:26.027172+0000","last_fullsized":"2024-03-31T15:56:26.027172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041125+0000","last_change":"2024-03-31T15:56:26.035712+0000","last_active":"2024-03-31T15:56:26.041125+0000","last_peered":"2024-03-31T15:56:26.041125+0000","last_clean":"2024-03-31T15:56:26.041125+0000","last_became_active":"2024-03-31T15:56:26.034040+0000","last_became_peered":"2024-03-31T15:56:26.034040+0000","last_unstale":"2024-03-31T15:56:26.041125+0000","last_undegraded":"2024-03-31T15:56:26.041125+0000","last_fullsized":"2024-03-31T15:56:26.041125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041511+0000","last_change":"2024-03-31T15:56:26.036984+0000","last_active":"2024-03-31T15:56:26.041511+0000","last_peered":"2024-03-31T15:56:26.041511+0000","last_clean":"2024-03-31T15:56:26.041511+0000","last_became_active":"2024-03-31T15:56:26.036674+0000","last_became_peered":"2024-03-31T15:56:26.036674+0000","last_unstale":"2024-03-31T15:56:26.041511+0000","last_undegraded":"2024-03-31T15:56:26.041511+0000","last_fullsized":"2024-03-31T15:56:26.041511+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.047955+0000","last_change":"2024-03-31T15:56:26.042322+0000","last_active":"2024-03-31T15:56:26.047955+0000","last_peered":"2024-03-31T15:56:26.047955+0000","last_clean":"2024-03-31T15:56:26.047955+0000","last_became_active":"2024-03-31T15:56:26.041942+0000","last_became_peered":"2024-03-31T15:56:26.041942+0000","last_unstale":"2024-03-31T15:56:26.047955+0000","last_undegraded":"2024-03-31T15:56:26.047955+0000","last_fullsized":"2024-03-31T15:56:26.047955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022096+0000","last_change":"2024-03-31T15:56:26.022096+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022096+0000","last_undegraded":"2024-03-31T15:56:26.022096+0000","last_fullsized":"2024-03-31T15:56:26.022096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.043837+0000","last_change":"2024-03-31T15:56:26.041462+0000","last_active":"2024-03-31T15:56:26.043837+0000","last_peered":"2024-03-31T15:56:26.043837+0000","last_clean":"2024-03-31T15:56:26.043837+0000","last_became_active":"2024-03-31T15:56:26.041184+0000","last_became_peered":"2024-03-31T15:56:26.041184+0000","last_unstale":"2024-03-31T15:56:26.043837+0000","last_undegraded":"2024-03-31T15:56:26.043837+0000","last_fullsized":"2024-03-31T15:56:26.043837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-03-31T15:56:25.016293+0000","last_change":"2024-03-31T15:56:25.016293+0000","last_active":"2024-03-31T15:56:25.006583+0000","last_peered":"2024-03-31T15:56:25.006583+0000","last_clean":"2024-03-31T15:56:25.006583+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:25.016293+0000","last_undegraded":"2024-03-31T15:56:25.016293+0000","last_fullsized":"2024-03-31T15:56:25.016293+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-31T15:56:26.009477+0000","last_change":"2024-03-31T15:56:26.009477+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.009477+0000","last_undegraded":"2024-03-31T15:56:26.009477+0000","last_fullsized":"2024-03-31T15:56:26.009477+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045156+0000","last_change":"2024-03-31T15:56:26.039648+0000","last_active":"2024-03-31T15:56:26.045156+0000","last_peered":"2024-03-31T15:56:26.045156+0000","last_clean":"2024-03-31T15:56:26.045156+0000","last_became_active":"2024-03-31T15:56:26.039276+0000","last_became_peered":"2024-03-31T15:56:26.039276+0000","last_unstale":"2024-03-31T15:56:26.045156+0000","last_undegraded":"2024-03-31T15:56:26.045156+0000","last_fullsized":"2024-03-31T15:56:26.045156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":231,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-31T15:56:23.012384+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:23.012384+0000","last_peered":"2024-03-31T15:56:23.012384+0000","last_clean":"2024-03-31T15:56:23.012384+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:23.012384+0000","last_undegraded":"2024-03-31T15:56:23.012384+0000","last_fullsized":"2024-03-31T15:56:23.012384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":66,"acting":66,"num_store_stats":5},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":58,"acting":58,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":273822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":870849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":602112,"data_stored":267613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":267613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1196032,"data_stored":857981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:56:31.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:56:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.304+0000 7f5cb6200640 1 Processor -- start 2024-03-31T15:56:31.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.304+0000 7f5cb6200640 1 -- start start 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5cb6200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5cb006c7b0 0x7f5cb006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5cb6200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb015f780 con 0x7f5cb015e1b0 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5cb6200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb006da20 con 0x7f5cb006d5f0 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5cb6200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5cb01622b0 con 0x7f5cb006c7b0 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5caf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5cb015e1b0 0x7f5cb0162740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50444/0 (socket says 172.21.15.90:50444) 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5caf400640 1 -- 172.21.15.90:0/859251195 learned_addr learned my addr 172.21.15.90:0/859251195 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:31.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5ca7e00640 1 --2- 172.21.15.90:0/859251195 >> v2:172.21.15.90:3300/0 conn(0x7f5cb006c7b0 0x7f5cb006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5ca7e00640 1 -- 172.21.15.90:0/859251195 >> v1:172.21.15.102:6789/0 conn(0x7f5cb006d5f0 legacy=0x7f5cb015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5ca7e00640 1 -- 172.21.15.90:0/859251195 >> v1:172.21.15.90:6789/0 conn(0x7f5cb015e1b0 legacy=0x7f5cb0162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5ca7e00640 1 -- 172.21.15.90:0/859251195 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb015f1c0 con 0x7f5cb006c7b0 2024-03-31T15:56:31.301 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.308+0000 7f5caea00640 1 -- 172.21.15.90:0/859251195 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3916465419 0 0) 0x7f5cb015f780 con 0x7f5cb015e1b0 2024-03-31T15:56:31.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5ca7e00640 1 --2- 172.21.15.90:0/859251195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb006c7b0 0x7f5cb006d0b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5c980097c0 tx=0x7f5c9801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba5fd626ea503fc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5caea00640 1 -- 172.21.15.90:0/859251195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c980b1070 con 0x7f5cb006c7b0 2024-03-31T15:56:31.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5caea00640 1 -- 172.21.15.90:0/859251195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c980a8050 con 0x7f5cb006c7b0 2024-03-31T15:56:31.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5caea00640 1 -- 172.21.15.90:0/859251195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c980ac860 con 0x7f5cb006c7b0 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- 172.21.15.90:0/859251195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb006c7b0 msgr2=0x7f5cb006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 --2- 172.21.15.90:0/859251195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb006c7b0 0x7f5cb006d0b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5c980097c0 tx=0x7f5c9801c3e0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- 172.21.15.90:0/859251195 shutdown_connections 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 --2- 172.21.15.90:0/859251195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb006c7b0 0x7f5cb006d0b0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- 172.21.15.90:0/859251195 >> 172.21.15.90:0/859251195 conn(0x7f5cb009aa40 msgr2=0x7f5cb009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- 172.21.15.90:0/859251195 shutdown_connections 2024-03-31T15:56:31.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- 172.21.15.90:0/859251195 wait complete. 2024-03-31T15:56:31.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 Processor -- start 2024-03-31T15:56:31.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- start start 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 0x7f5cb00cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb006d5f0 0x7f5cb00c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb00ce7a0 con 0x7f5cb015e1b0 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb00d2a90 con 0x7f5cb006d5f0 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cb6200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb00ce920 con 0x7f5cb006c7b0 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cafe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41934/0 (socket says 172.21.15.90:41934) 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.312+0000 7f5cafe00640 1 -- 172.21.15.90:0/523974050 learned_addr learned my addr 172.21.15.90:0/523974050 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7e00640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb006d5f0 0x7f5cb00c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5caf400640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 0x7f5cb00cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 -- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 msgr2=0x7f5cb00cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 0x7f5cb00cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 -- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb006d5f0 msgr2=0x7f5cb00c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb006d5f0 0x7f5cb00c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 -- 172.21.15.90:0/523974050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb00c7dd0 con 0x7f5cb015e1b0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5caf400640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 0x7f5cb00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cafe00640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800c770 tx=0x7f5ca800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8003b80 con 0x7f5cb015e1b0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ca8003d20 con 0x7f5cb015e1b0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb00c80b0 con 0x7f5cb015e1b0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca800aa30 con 0x7f5cb015e1b0 2024-03-31T15:56:31.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb00cbb00 con 0x7f5cb015e1b0 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb00cbdf0 con 0x7f5cb015e1b0 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ca800abd0 con 0x7f5cb015e1b0 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 0x7f5c8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.316+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f5ca8086b10 con 0x7f5cb015e1b0 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.320+0000 7f5caf400640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 0x7f5c8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.320+0000 7f5caf400640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 0x7f5c8c06a840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0000f70 tx=0x7f5ca0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.320+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5ca800ce20 con 0x7f5cb015e1b0 2024-03-31T15:56:31.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.580+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5cb00c8560 con 0x7f5c8c068380 2024-03-31T15:56:31.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.588+0000 7f5ca7400640 1 -- 172.21.15.90:0/523974050 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249206 (secure 0 0 0) 0x7f5cb00c8560 con 0x7f5c8c068380 2024-03-31T15:56:31.583 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:31.584 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:56:31.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 msgr2=0x7f5c8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 0x7f5c8c06a840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0000f70 tx=0x7f5ca0009040 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 msgr2=0x7f5cb00c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800c770 tx=0x7f5ca800cc30 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 shutdown_connections 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5c8c068380 0x7f5c8c06a840 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5cb015e1b0 0x7f5cb00c77e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cb006d5f0 0x7f5cb00c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 --2- 172.21.15.90:0/523974050 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5cb006c7b0 0x7f5cb00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 >> 172.21.15.90:0/523974050 conn(0x7f5cb009aa40 msgr2=0x7f5cb00705d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 shutdown_connections 2024-03-31T15:56:31.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.592+0000 7f5cb6200640 1 -- 172.21.15.90:0/523974050 wait complete. 2024-03-31T15:56:31.613 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-31T15:56:31.255055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590514,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":180,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222312,"kb_used_data":6140,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752408,"statfs":{"total":773094113280,"available":772866465792,"internally_reserved":0,"allocated":6287360,"data_stored":3459774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001246"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732549+0000","last_change":"2024-03-31T15:56:27.060874+0000","last_active":"2024-03-31T15:56:28.732549+0000","last_peered":"2024-03-31T15:56:28.732549+0000","last_clean":"2024-03-31T15:56:28.732549+0000","last_became_active":"2024-03-31T15:56:27.060630+0000","last_became_peered":"2024-03-31T15:56:27.060630+0000","last_unstale":"2024-03-31T15:56:28.732549+0000","last_undegraded":"2024-03-31T15:56:28.732549+0000","last_fullsized":"2024-03-31T15:56:28.732549+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:56:54.783818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734232+0000","last_change":"2024-03-31T15:56:27.040029+0000","last_active":"2024-03-31T15:56:28.734232+0000","last_peered":"2024-03-31T15:56:28.734232+0000","last_clean":"2024-03-31T15:56:28.734232+0000","last_became_active":"2024-03-31T15:56:27.036544+0000","last_became_peered":"2024-03-31T15:56:27.036544+0000","last_unstale":"2024-03-31T15:56:28.734232+0000","last_undegraded":"2024-03-31T15:56:28.734232+0000","last_fullsized":"2024-03-31T15:56:28.734232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:57.935165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048573+0000","last_change":"2024-03-31T15:56:27.040254+0000","last_active":"2024-03-31T15:56:28.048573+0000","last_peered":"2024-03-31T15:56:28.048573+0000","last_clean":"2024-03-31T15:56:28.048573+0000","last_became_active":"2024-03-31T15:56:27.039839+0000","last_became_peered":"2024-03-31T15:56:27.039839+0000","last_unstale":"2024-03-31T15:56:28.048573+0000","last_undegraded":"2024-03-31T15:56:28.048573+0000","last_fullsized":"2024-03-31T15:56:28.048573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:38.034420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021551+0000","last_change":"2024-03-31T15:56:26.021551+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021551+0000","last_undegraded":"2024-03-31T15:56:26.021551+0000","last_fullsized":"2024-03-31T15:56:26.021551+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731537+0000","last_change":"2024-03-31T15:56:27.057953+0000","last_active":"2024-03-31T15:56:28.731537+0000","last_peered":"2024-03-31T15:56:28.731537+0000","last_clean":"2024-03-31T15:56:28.731537+0000","last_became_active":"2024-03-31T15:56:27.057680+0000","last_became_peered":"2024-03-31T15:56:27.057680+0000","last_unstale":"2024-03-31T15:56:28.731537+0000","last_undegraded":"2024-03-31T15:56:28.731537+0000","last_fullsized":"2024-03-31T15:56:28.731537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:41:20.883103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025926+0000","last_change":"2024-03-31T15:56:26.025926+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025926+0000","last_undegraded":"2024-03-31T15:56:26.025926+0000","last_fullsized":"2024-03-31T15:56:26.025926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028615+0000","last_change":"2024-03-31T15:56:26.028615+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028615+0000","last_undegraded":"2024-03-31T15:56:26.028615+0000","last_fullsized":"2024-03-31T15:56:26.028615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733707+0000","last_change":"2024-03-31T15:56:27.040713+0000","last_active":"2024-03-31T15:56:28.733707+0000","last_peered":"2024-03-31T15:56:28.733707+0000","last_clean":"2024-03-31T15:56:28.733707+0000","last_became_active":"2024-03-31T15:56:27.040439+0000","last_became_peered":"2024-03-31T15:56:27.040439+0000","last_unstale":"2024-03-31T15:56:28.733707+0000","last_undegraded":"2024-03-31T15:56:28.733707+0000","last_fullsized":"2024-03-31T15:56:28.733707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026552+0000","last_change":"2024-03-31T15:56:26.026552+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026552+0000","last_undegraded":"2024-03-31T15:56:26.026552+0000","last_fullsized":"2024-03-31T15:56:26.026552+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022133+0000","last_change":"2024-03-31T15:56:26.022133+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022133+0000","last_undegraded":"2024-03-31T15:56:26.022133+0000","last_fullsized":"2024-03-31T15:56:26.022133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022163+0000","last_change":"2024-03-31T15:56:26.022163+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022163+0000","last_undegraded":"2024-03-31T15:56:26.022163+0000","last_fullsized":"2024-03-31T15:56:26.022163+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021075+0000","last_change":"2024-03-31T15:56:26.021075+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021075+0000","last_undegraded":"2024-03-31T15:56:26.021075+0000","last_fullsized":"2024-03-31T15:56:26.021075+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028321+0000","last_change":"2024-03-31T15:56:26.028321+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028321+0000","last_undegraded":"2024-03-31T15:56:26.028321+0000","last_fullsized":"2024-03-31T15:56:26.028321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023949+0000","last_change":"2024-03-31T15:56:26.023949+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023949+0000","last_undegraded":"2024-03-31T15:56:26.023949+0000","last_fullsized":"2024-03-31T15:56:26.023949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027463+0000","last_change":"2024-03-31T15:56:26.027463+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027463+0000","last_undegraded":"2024-03-31T15:56:26.027463+0000","last_fullsized":"2024-03-31T15:56:26.027463+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025451+0000","last_change":"2024-03-31T15:56:26.025451+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025451+0000","last_undegraded":"2024-03-31T15:56:26.025451+0000","last_fullsized":"2024-03-31T15:56:26.025451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734398+0000","last_change":"2024-03-31T15:56:27.041513+0000","last_active":"2024-03-31T15:56:28.734398+0000","last_peered":"2024-03-31T15:56:28.734398+0000","last_clean":"2024-03-31T15:56:28.734398+0000","last_became_active":"2024-03-31T15:56:27.041210+0000","last_became_peered":"2024-03-31T15:56:27.041210+0000","last_unstale":"2024-03-31T15:56:28.734398+0000","last_undegraded":"2024-03-31T15:56:28.734398+0000","last_fullsized":"2024-03-31T15:56:28.734398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:55.146299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048591+0000","last_change":"2024-03-31T15:56:27.049282+0000","last_active":"2024-03-31T15:56:28.048591+0000","last_peered":"2024-03-31T15:56:28.048591+0000","last_clean":"2024-03-31T15:56:28.048591+0000","last_became_active":"2024-03-31T15:56:27.048536+0000","last_became_peered":"2024-03-31T15:56:27.048536+0000","last_unstale":"2024-03-31T15:56:28.048591+0000","last_undegraded":"2024-03-31T15:56:28.048591+0000","last_fullsized":"2024-03-31T15:56:28.048591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:49:37.697269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732485+0000","last_change":"2024-03-31T15:56:27.060292+0000","last_active":"2024-03-31T15:56:28.732485+0000","last_peered":"2024-03-31T15:56:28.732485+0000","last_clean":"2024-03-31T15:56:28.732485+0000","last_became_active":"2024-03-31T15:56:27.060044+0000","last_became_peered":"2024-03-31T15:56:27.060044+0000","last_unstale":"2024-03-31T15:56:28.732485+0000","last_undegraded":"2024-03-31T15:56:28.732485+0000","last_fullsized":"2024-03-31T15:56:28.732485+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:03.617564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734311+0000","last_change":"2024-03-31T15:56:27.059752+0000","last_active":"2024-03-31T15:56:28.734311+0000","last_peered":"2024-03-31T15:56:28.734311+0000","last_clean":"2024-03-31T15:56:28.734311+0000","last_became_active":"2024-03-31T15:56:27.059504+0000","last_became_peered":"2024-03-31T15:56:27.059504+0000","last_unstale":"2024-03-31T15:56:28.734311+0000","last_undegraded":"2024-03-31T15:56:28.734311+0000","last_fullsized":"2024-03-31T15:56:28.734311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:03.532453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048464+0000","last_change":"2024-03-31T15:56:27.030882+0000","last_active":"2024-03-31T15:56:28.048464+0000","last_peered":"2024-03-31T15:56:28.048464+0000","last_clean":"2024-03-31T15:56:28.048464+0000","last_became_active":"2024-03-31T15:56:27.030222+0000","last_became_peered":"2024-03-31T15:56:27.030222+0000","last_unstale":"2024-03-31T15:56:28.048464+0000","last_undegraded":"2024-03-31T15:56:28.048464+0000","last_fullsized":"2024-03-31T15:56:28.048464+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:25.459268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733876+0000","last_change":"2024-03-31T15:56:27.053182+0000","last_active":"2024-03-31T15:56:28.733876+0000","last_peered":"2024-03-31T15:56:28.733876+0000","last_clean":"2024-03-31T15:56:28.733876+0000","last_became_active":"2024-03-31T15:56:27.052882+0000","last_became_peered":"2024-03-31T15:56:27.052882+0000","last_unstale":"2024-03-31T15:56:28.733876+0000","last_undegraded":"2024-03-31T15:56:28.733876+0000","last_fullsized":"2024-03-31T15:56:28.733876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047955+0000","last_change":"2024-03-31T15:56:27.037326+0000","last_active":"2024-03-31T15:56:28.047955+0000","last_peered":"2024-03-31T15:56:28.047955+0000","last_clean":"2024-03-31T15:56:28.047955+0000","last_became_active":"2024-03-31T15:56:27.036842+0000","last_became_peered":"2024-03-31T15:56:27.036842+0000","last_unstale":"2024-03-31T15:56:28.047955+0000","last_undegraded":"2024-03-31T15:56:28.047955+0000","last_fullsized":"2024-03-31T15:56:28.047955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:13.092644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026315+0000","last_change":"2024-03-31T15:56:26.026315+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026315+0000","last_undegraded":"2024-03-31T15:56:26.026315+0000","last_fullsized":"2024-03-31T15:56:26.026315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732648+0000","last_change":"2024-03-31T15:56:27.058506+0000","last_active":"2024-03-31T15:56:28.732648+0000","last_peered":"2024-03-31T15:56:28.732648+0000","last_clean":"2024-03-31T15:56:28.732648+0000","last_became_active":"2024-03-31T15:56:27.058252+0000","last_became_peered":"2024-03-31T15:56:27.058252+0000","last_unstale":"2024-03-31T15:56:28.732648+0000","last_undegraded":"2024-03-31T15:56:28.732648+0000","last_fullsized":"2024-03-31T15:56:28.732648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:03.064057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026205+0000","last_change":"2024-03-31T15:56:26.026205+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026205+0000","last_undegraded":"2024-03-31T15:56:26.026205+0000","last_fullsized":"2024-03-31T15:56:26.026205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031242+0000","last_change":"2024-03-31T15:56:26.031242+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031242+0000","last_undegraded":"2024-03-31T15:56:26.031242+0000","last_fullsized":"2024-03-31T15:56:26.031242+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732586+0000","last_change":"2024-03-31T15:56:27.058392+0000","last_active":"2024-03-31T15:56:28.732586+0000","last_peered":"2024-03-31T15:56:28.732586+0000","last_clean":"2024-03-31T15:56:28.732586+0000","last_became_active":"2024-03-31T15:56:27.058103+0000","last_became_peered":"2024-03-31T15:56:27.058103+0000","last_unstale":"2024-03-31T15:56:28.732586+0000","last_undegraded":"2024-03-31T15:56:28.732586+0000","last_fullsized":"2024-03-31T15:56:28.732586+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:38.875392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734201+0000","last_change":"2024-03-31T15:56:27.041576+0000","last_active":"2024-03-31T15:56:28.734201+0000","last_peered":"2024-03-31T15:56:28.734201+0000","last_clean":"2024-03-31T15:56:28.734201+0000","last_became_active":"2024-03-31T15:56:27.041327+0000","last_became_peered":"2024-03-31T15:56:27.041327+0000","last_unstale":"2024-03-31T15:56:28.734201+0000","last_undegraded":"2024-03-31T15:56:28.734201+0000","last_fullsized":"2024-03-31T15:56:28.734201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:39.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048359+0000","last_change":"2024-03-31T15:56:27.034473+0000","last_active":"2024-03-31T15:56:28.048359+0000","last_peered":"2024-03-31T15:56:28.048359+0000","last_clean":"2024-03-31T15:56:28.048359+0000","last_became_active":"2024-03-31T15:56:27.034209+0000","last_became_peered":"2024-03-31T15:56:27.034209+0000","last_unstale":"2024-03-31T15:56:28.048359+0000","last_undegraded":"2024-03-31T15:56:28.048359+0000","last_fullsized":"2024-03-31T15:56:28.048359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:45.169835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731505+0000","last_change":"2024-03-31T15:56:27.058056+0000","last_active":"2024-03-31T15:56:28.731505+0000","last_peered":"2024-03-31T15:56:28.731505+0000","last_clean":"2024-03-31T15:56:28.731505+0000","last_became_active":"2024-03-31T15:56:27.057786+0000","last_became_peered":"2024-03-31T15:56:27.057786+0000","last_unstale":"2024-03-31T15:56:28.731505+0000","last_undegraded":"2024-03-31T15:56:28.731505+0000","last_fullsized":"2024-03-31T15:56:28.731505+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:04:01.295156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024954+0000","last_change":"2024-03-31T15:56:26.024954+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024954+0000","last_undegraded":"2024-03-31T15:56:26.024954+0000","last_fullsized":"2024-03-31T15:56:26.024954+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031116+0000","last_change":"2024-03-31T15:56:26.031116+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031116+0000","last_undegraded":"2024-03-31T15:56:26.031116+0000","last_fullsized":"2024-03-31T15:56:26.031116+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044934+0000","last_change":"2024-03-31T15:56:26.043587+0000","last_active":"2024-03-31T15:56:26.044934+0000","last_peered":"2024-03-31T15:56:26.044934+0000","last_clean":"2024-03-31T15:56:26.044934+0000","last_became_active":"2024-03-31T15:56:26.043321+0000","last_became_peered":"2024-03-31T15:56:26.043321+0000","last_unstale":"2024-03-31T15:56:26.044934+0000","last_undegraded":"2024-03-31T15:56:26.044934+0000","last_fullsized":"2024-03-31T15:56:26.044934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027162+0000","last_change":"2024-03-31T15:56:26.027162+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027162+0000","last_undegraded":"2024-03-31T15:56:26.027162+0000","last_fullsized":"2024-03-31T15:56:26.027162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734183+0000","last_change":"2024-03-31T15:56:26.029688+0000","last_active":"2024-03-31T15:56:28.734183+0000","last_peered":"2024-03-31T15:56:28.734183+0000","last_clean":"2024-03-31T15:56:28.734183+0000","last_became_active":"2024-03-31T15:56:26.029396+0000","last_became_peered":"2024-03-31T15:56:26.029396+0000","last_unstale":"2024-03-31T15:56:28.734183+0000","last_undegraded":"2024-03-31T15:56:28.734183+0000","last_fullsized":"2024-03-31T15:56:28.734183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:52:46.291141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026382+0000","last_change":"2024-03-31T15:56:26.026382+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026382+0000","last_undegraded":"2024-03-31T15:56:26.026382+0000","last_fullsized":"2024-03-31T15:56:26.026382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045804+0000","last_change":"2024-03-31T15:56:26.044595+0000","last_active":"2024-03-31T15:56:26.045804+0000","last_peered":"2024-03-31T15:56:26.045804+0000","last_clean":"2024-03-31T15:56:26.045804+0000","last_became_active":"2024-03-31T15:56:26.044321+0000","last_became_peered":"2024-03-31T15:56:26.044321+0000","last_unstale":"2024-03-31T15:56:26.045804+0000","last_undegraded":"2024-03-31T15:56:26.045804+0000","last_fullsized":"2024-03-31T15:56:26.045804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027580+0000","last_change":"2024-03-31T15:56:26.027580+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027580+0000","last_undegraded":"2024-03-31T15:56:26.027580+0000","last_fullsized":"2024-03-31T15:56:26.027580+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.038619+0000","last_change":"2024-03-31T15:56:26.035102+0000","last_active":"2024-03-31T15:56:26.038619+0000","last_peered":"2024-03-31T15:56:26.038619+0000","last_clean":"2024-03-31T15:56:26.038619+0000","last_became_active":"2024-03-31T15:56:26.034821+0000","last_became_peered":"2024-03-31T15:56:26.034821+0000","last_unstale":"2024-03-31T15:56:26.038619+0000","last_undegraded":"2024-03-31T15:56:26.038619+0000","last_fullsized":"2024-03-31T15:56:26.038619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733891+0000","last_change":"2024-03-31T15:56:27.053152+0000","last_active":"2024-03-31T15:56:28.733891+0000","last_peered":"2024-03-31T15:56:28.733891+0000","last_clean":"2024-03-31T15:56:28.733891+0000","last_became_active":"2024-03-31T15:56:27.052870+0000","last_became_peered":"2024-03-31T15:56:27.052870+0000","last_unstale":"2024-03-31T15:56:28.733891+0000","last_undegraded":"2024-03-31T15:56:28.733891+0000","last_fullsized":"2024-03-31T15:56:28.733891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:53.273112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042800+0000","last_change":"2024-03-31T15:56:26.035499+0000","last_active":"2024-03-31T15:56:26.042800+0000","last_peered":"2024-03-31T15:56:26.042800+0000","last_clean":"2024-03-31T15:56:26.042800+0000","last_became_active":"2024-03-31T15:56:26.035216+0000","last_became_peered":"2024-03-31T15:56:26.035216+0000","last_unstale":"2024-03-31T15:56:26.042800+0000","last_undegraded":"2024-03-31T15:56:26.042800+0000","last_fullsized":"2024-03-31T15:56:26.042800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734108+0000","last_change":"2024-03-31T15:56:27.055018+0000","last_active":"2024-03-31T15:56:28.734108+0000","last_peered":"2024-03-31T15:56:28.734108+0000","last_clean":"2024-03-31T15:56:28.734108+0000","last_became_active":"2024-03-31T15:56:27.054765+0000","last_became_peered":"2024-03-31T15:56:27.054765+0000","last_unstale":"2024-03-31T15:56:28.734108+0000","last_undegraded":"2024-03-31T15:56:28.734108+0000","last_fullsized":"2024-03-31T15:56:28.734108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:49.516522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044103+0000","last_change":"2024-03-31T15:56:26.040614+0000","last_active":"2024-03-31T15:56:26.044103+0000","last_peered":"2024-03-31T15:56:26.044103+0000","last_clean":"2024-03-31T15:56:26.044103+0000","last_became_active":"2024-03-31T15:56:26.040371+0000","last_became_peered":"2024-03-31T15:56:26.040371+0000","last_unstale":"2024-03-31T15:56:26.044103+0000","last_undegraded":"2024-03-31T15:56:26.044103+0000","last_fullsized":"2024-03-31T15:56:26.044103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025656+0000","last_change":"2024-03-31T15:56:26.025656+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025656+0000","last_undegraded":"2024-03-31T15:56:26.025656+0000","last_fullsized":"2024-03-31T15:56:26.025656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041171+0000","last_change":"2024-03-31T15:56:26.037304+0000","last_active":"2024-03-31T15:56:26.041171+0000","last_peered":"2024-03-31T15:56:26.041171+0000","last_clean":"2024-03-31T15:56:26.041171+0000","last_became_active":"2024-03-31T15:56:26.037038+0000","last_became_peered":"2024-03-31T15:56:26.037038+0000","last_unstale":"2024-03-31T15:56:26.041171+0000","last_undegraded":"2024-03-31T15:56:26.041171+0000","last_fullsized":"2024-03-31T15:56:26.041171+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:52.264596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733928+0000","last_change":"2024-03-31T15:56:27.043180+0000","last_active":"2024-03-31T15:56:28.733928+0000","last_peered":"2024-03-31T15:56:28.733928+0000","last_clean":"2024-03-31T15:56:28.733928+0000","last_became_active":"2024-03-31T15:56:27.042692+0000","last_became_peered":"2024-03-31T15:56:27.042692+0000","last_unstale":"2024-03-31T15:56:28.733928+0000","last_undegraded":"2024-03-31T15:56:28.733928+0000","last_fullsized":"2024-03-31T15:56:28.733928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:21:35.971485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045526+0000","last_change":"2024-03-31T15:56:26.040038+0000","last_active":"2024-03-31T15:56:26.045526+0000","last_peered":"2024-03-31T15:56:26.045526+0000","last_clean":"2024-03-31T15:56:26.045526+0000","last_became_active":"2024-03-31T15:56:26.039654+0000","last_became_peered":"2024-03-31T15:56:26.039654+0000","last_unstale":"2024-03-31T15:56:26.045526+0000","last_undegraded":"2024-03-31T15:56:26.045526+0000","last_fullsized":"2024-03-31T15:56:26.045526+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023782+0000","last_change":"2024-03-31T15:56:26.023782+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023782+0000","last_undegraded":"2024-03-31T15:56:26.023782+0000","last_fullsized":"2024-03-31T15:56:26.023782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049029+0000","last_change":"2024-03-31T15:56:26.041995+0000","last_active":"2024-03-31T15:56:28.049029+0000","last_peered":"2024-03-31T15:56:28.049029+0000","last_clean":"2024-03-31T15:56:28.049029+0000","last_became_active":"2024-03-31T15:56:26.041698+0000","last_became_peered":"2024-03-31T15:56:26.041698+0000","last_unstale":"2024-03-31T15:56:28.049029+0000","last_undegraded":"2024-03-31T15:56:28.049029+0000","last_fullsized":"2024-03-31T15:56:28.049029+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:03.382018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049076+0000","last_change":"2024-03-31T15:56:27.032991+0000","last_active":"2024-03-31T15:56:28.049076+0000","last_peered":"2024-03-31T15:56:28.049076+0000","last_clean":"2024-03-31T15:56:28.049076+0000","last_became_active":"2024-03-31T15:56:27.032057+0000","last_became_peered":"2024-03-31T15:56:27.032057+0000","last_unstale":"2024-03-31T15:56:28.049076+0000","last_undegraded":"2024-03-31T15:56:28.049076+0000","last_fullsized":"2024-03-31T15:56:28.049076+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734295+0000","last_change":"2024-03-31T15:56:26.034420+0000","last_active":"2024-03-31T15:56:28.734295+0000","last_peered":"2024-03-31T15:56:28.734295+0000","last_clean":"2024-03-31T15:56:28.734295+0000","last_became_active":"2024-03-31T15:56:26.034145+0000","last_became_peered":"2024-03-31T15:56:26.034145+0000","last_unstale":"2024-03-31T15:56:28.734295+0000","last_undegraded":"2024-03-31T15:56:28.734295+0000","last_fullsized":"2024-03-31T15:56:28.734295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:18.044685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023933+0000","last_change":"2024-03-31T15:56:26.023933+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023933+0000","last_undegraded":"2024-03-31T15:56:26.023933+0000","last_fullsized":"2024-03-31T15:56:26.023933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048878+0000","last_change":"2024-03-31T15:56:26.044785+0000","last_active":"2024-03-31T15:56:28.048878+0000","last_peered":"2024-03-31T15:56:28.048878+0000","last_clean":"2024-03-31T15:56:28.048878+0000","last_became_active":"2024-03-31T15:56:26.044515+0000","last_became_peered":"2024-03-31T15:56:26.044515+0000","last_unstale":"2024-03-31T15:56:28.048878+0000","last_undegraded":"2024-03-31T15:56:28.048878+0000","last_fullsized":"2024-03-31T15:56:28.048878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:17.779102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048748+0000","last_change":"2024-03-31T15:56:27.032521+0000","last_active":"2024-03-31T15:56:28.048748+0000","last_peered":"2024-03-31T15:56:28.048748+0000","last_clean":"2024-03-31T15:56:28.048748+0000","last_became_active":"2024-03-31T15:56:27.032235+0000","last_became_peered":"2024-03-31T15:56:27.032235+0000","last_unstale":"2024-03-31T15:56:28.048748+0000","last_undegraded":"2024-03-31T15:56:28.048748+0000","last_fullsized":"2024-03-31T15:56:28.048748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045554+0000","last_change":"2024-03-31T15:56:26.041173+0000","last_active":"2024-03-31T15:56:26.045554+0000","last_peered":"2024-03-31T15:56:26.045554+0000","last_clean":"2024-03-31T15:56:26.045554+0000","last_became_active":"2024-03-31T15:56:26.040901+0000","last_became_peered":"2024-03-31T15:56:26.040901+0000","last_unstale":"2024-03-31T15:56:26.045554+0000","last_undegraded":"2024-03-31T15:56:26.045554+0000","last_fullsized":"2024-03-31T15:56:26.045554+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022785+0000","last_change":"2024-03-31T15:56:26.022785+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022785+0000","last_undegraded":"2024-03-31T15:56:26.022785+0000","last_fullsized":"2024-03-31T15:56:26.022785+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733993+0000","last_change":"2024-03-31T15:56:26.035713+0000","last_active":"2024-03-31T15:56:28.733993+0000","last_peered":"2024-03-31T15:56:28.733993+0000","last_clean":"2024-03-31T15:56:28.733993+0000","last_became_active":"2024-03-31T15:56:26.035393+0000","last_became_peered":"2024-03-31T15:56:26.035393+0000","last_unstale":"2024-03-31T15:56:28.733993+0000","last_undegraded":"2024-03-31T15:56:28.733993+0000","last_fullsized":"2024-03-31T15:56:28.733993+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:10.726676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048441+0000","last_change":"2024-03-31T15:56:27.041906+0000","last_active":"2024-03-31T15:56:28.048441+0000","last_peered":"2024-03-31T15:56:28.048441+0000","last_clean":"2024-03-31T15:56:28.048441+0000","last_became_active":"2024-03-31T15:56:27.041609+0000","last_became_peered":"2024-03-31T15:56:27.041609+0000","last_unstale":"2024-03-31T15:56:28.048441+0000","last_undegraded":"2024-03-31T15:56:28.048441+0000","last_fullsized":"2024-03-31T15:56:28.048441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:53:45.939898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048116+0000","last_change":"2024-03-31T15:56:26.041375+0000","last_active":"2024-03-31T15:56:26.048116+0000","last_peered":"2024-03-31T15:56:26.048116+0000","last_clean":"2024-03-31T15:56:26.048116+0000","last_became_active":"2024-03-31T15:56:26.041158+0000","last_became_peered":"2024-03-31T15:56:26.041158+0000","last_unstale":"2024-03-31T15:56:26.048116+0000","last_undegraded":"2024-03-31T15:56:26.048116+0000","last_fullsized":"2024-03-31T15:56:26.048116+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024021+0000","last_change":"2024-03-31T15:56:26.024021+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024021+0000","last_undegraded":"2024-03-31T15:56:26.024021+0000","last_fullsized":"2024-03-31T15:56:26.024021+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041150+0000","last_change":"2024-03-31T15:56:26.037434+0000","last_active":"2024-03-31T15:56:26.041150+0000","last_peered":"2024-03-31T15:56:26.041150+0000","last_clean":"2024-03-31T15:56:26.041150+0000","last_became_active":"2024-03-31T15:56:26.036753+0000","last_became_peered":"2024-03-31T15:56:26.036753+0000","last_unstale":"2024-03-31T15:56:26.041150+0000","last_undegraded":"2024-03-31T15:56:26.041150+0000","last_fullsized":"2024-03-31T15:56:26.041150+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.472784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047988+0000","last_change":"2024-03-31T15:56:27.037112+0000","last_active":"2024-03-31T15:56:28.047988+0000","last_peered":"2024-03-31T15:56:28.047988+0000","last_clean":"2024-03-31T15:56:28.047988+0000","last_became_active":"2024-03-31T15:56:27.036824+0000","last_became_peered":"2024-03-31T15:56:27.036824+0000","last_unstale":"2024-03-31T15:56:28.047988+0000","last_undegraded":"2024-03-31T15:56:28.047988+0000","last_fullsized":"2024-03-31T15:56:28.047988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:44:17.091367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047288+0000","last_change":"2024-03-31T15:56:26.038121+0000","last_active":"2024-03-31T15:56:28.047288+0000","last_peered":"2024-03-31T15:56:28.047288+0000","last_clean":"2024-03-31T15:56:28.047288+0000","last_became_active":"2024-03-31T15:56:26.037834+0000","last_became_peered":"2024-03-31T15:56:26.037834+0000","last_unstale":"2024-03-31T15:56:28.047288+0000","last_undegraded":"2024-03-31T15:56:28.047288+0000","last_fullsized":"2024-03-31T15:56:28.047288+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:38.551620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048667+0000","last_change":"2024-03-31T15:56:27.038762+0000","last_active":"2024-03-31T15:56:28.048667+0000","last_peered":"2024-03-31T15:56:28.048667+0000","last_clean":"2024-03-31T15:56:28.048667+0000","last_became_active":"2024-03-31T15:56:27.038442+0000","last_became_peered":"2024-03-31T15:56:27.038442+0000","last_unstale":"2024-03-31T15:56:28.048667+0000","last_undegraded":"2024-03-31T15:56:28.048667+0000","last_fullsized":"2024-03-31T15:56:28.048667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:23.516599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734348+0000","last_change":"2024-03-31T15:56:26.034027+0000","last_active":"2024-03-31T15:56:28.734348+0000","last_peered":"2024-03-31T15:56:28.734348+0000","last_clean":"2024-03-31T15:56:28.734348+0000","last_became_active":"2024-03-31T15:56:26.033735+0000","last_became_peered":"2024-03-31T15:56:26.033735+0000","last_unstale":"2024-03-31T15:56:28.734348+0000","last_undegraded":"2024-03-31T15:56:28.734348+0000","last_fullsized":"2024-03-31T15:56:28.734348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:46.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027856+0000","last_change":"2024-03-31T15:56:26.027856+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027856+0000","last_undegraded":"2024-03-31T15:56:26.027856+0000","last_fullsized":"2024-03-31T15:56:26.027856+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042183+0000","last_change":"2024-03-31T15:56:26.039370+0000","last_active":"2024-03-31T15:56:26.042183+0000","last_peered":"2024-03-31T15:56:26.042183+0000","last_clean":"2024-03-31T15:56:26.042183+0000","last_became_active":"2024-03-31T15:56:26.038769+0000","last_became_peered":"2024-03-31T15:56:26.038769+0000","last_unstale":"2024-03-31T15:56:26.042183+0000","last_undegraded":"2024-03-31T15:56:26.042183+0000","last_fullsized":"2024-03-31T15:56:26.042183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:36.882522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734424+0000","last_change":"2024-03-31T15:56:27.040609+0000","last_active":"2024-03-31T15:56:28.734424+0000","last_peered":"2024-03-31T15:56:28.734424+0000","last_clean":"2024-03-31T15:56:28.734424+0000","last_became_active":"2024-03-31T15:56:27.040329+0000","last_became_peered":"2024-03-31T15:56:27.040329+0000","last_unstale":"2024-03-31T15:56:28.734424+0000","last_undegraded":"2024-03-31T15:56:28.734424+0000","last_fullsized":"2024-03-31T15:56:28.734424+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:25.079313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041058+0000","last_change":"2024-03-31T15:56:26.036191+0000","last_active":"2024-03-31T15:56:26.041058+0000","last_peered":"2024-03-31T15:56:26.041058+0000","last_clean":"2024-03-31T15:56:26.041058+0000","last_became_active":"2024-03-31T15:56:26.035873+0000","last_became_peered":"2024-03-31T15:56:26.035873+0000","last_unstale":"2024-03-31T15:56:26.041058+0000","last_undegraded":"2024-03-31T15:56:26.041058+0000","last_fullsized":"2024-03-31T15:56:26.041058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048941+0000","last_change":"2024-03-31T15:56:27.034376+0000","last_active":"2024-03-31T15:56:28.048941+0000","last_peered":"2024-03-31T15:56:28.048941+0000","last_clean":"2024-03-31T15:56:28.048941+0000","last_became_active":"2024-03-31T15:56:27.032545+0000","last_became_peered":"2024-03-31T15:56:27.032545+0000","last_unstale":"2024-03-31T15:56:28.048941+0000","last_undegraded":"2024-03-31T15:56:28.048941+0000","last_fullsized":"2024-03-31T15:56:28.048941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045774+0000","last_change":"2024-03-31T15:56:26.044352+0000","last_active":"2024-03-31T15:56:26.045774+0000","last_peered":"2024-03-31T15:56:26.045774+0000","last_clean":"2024-03-31T15:56:26.045774+0000","last_became_active":"2024-03-31T15:56:26.044071+0000","last_became_peered":"2024-03-31T15:56:26.044071+0000","last_unstale":"2024-03-31T15:56:26.045774+0000","last_undegraded":"2024-03-31T15:56:26.045774+0000","last_fullsized":"2024-03-31T15:56:26.045774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022357+0000","last_change":"2024-03-31T15:56:26.022357+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022357+0000","last_undegraded":"2024-03-31T15:56:26.022357+0000","last_fullsized":"2024-03-31T15:56:26.022357+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734169+0000","last_change":"2024-03-31T15:56:26.030253+0000","last_active":"2024-03-31T15:56:28.734169+0000","last_peered":"2024-03-31T15:56:28.734169+0000","last_clean":"2024-03-31T15:56:28.734169+0000","last_became_active":"2024-03-31T15:56:26.029995+0000","last_became_peered":"2024-03-31T15:56:26.029995+0000","last_unstale":"2024-03-31T15:56:28.734169+0000","last_undegraded":"2024-03-31T15:56:28.734169+0000","last_fullsized":"2024-03-31T15:56:28.734169+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:45:21.317263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025805+0000","last_change":"2024-03-31T15:56:26.025805+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025805+0000","last_undegraded":"2024-03-31T15:56:26.025805+0000","last_fullsized":"2024-03-31T15:56:26.025805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041491+0000","last_change":"2024-03-31T15:56:26.037979+0000","last_active":"2024-03-31T15:56:26.041491+0000","last_peered":"2024-03-31T15:56:26.041491+0000","last_clean":"2024-03-31T15:56:26.041491+0000","last_became_active":"2024-03-31T15:56:26.037707+0000","last_became_peered":"2024-03-31T15:56:26.037707+0000","last_unstale":"2024-03-31T15:56:26.041491+0000","last_undegraded":"2024-03-31T15:56:26.041491+0000","last_fullsized":"2024-03-31T15:56:26.041491+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026702+0000","last_change":"2024-03-31T15:56:26.026702+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026702+0000","last_undegraded":"2024-03-31T15:56:26.026702+0000","last_fullsized":"2024-03-31T15:56:26.026702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.036608+0000","last_change":"2024-03-31T15:56:26.030797+0000","last_active":"2024-03-31T15:56:26.036608+0000","last_peered":"2024-03-31T15:56:26.036608+0000","last_clean":"2024-03-31T15:56:26.036608+0000","last_became_active":"2024-03-31T15:56:26.030470+0000","last_became_peered":"2024-03-31T15:56:26.030470+0000","last_unstale":"2024-03-31T15:56:26.036608+0000","last_undegraded":"2024-03-31T15:56:26.036608+0000","last_fullsized":"2024-03-31T15:56:26.036608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025774+0000","last_change":"2024-03-31T15:56:26.025774+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025774+0000","last_undegraded":"2024-03-31T15:56:26.025774+0000","last_fullsized":"2024-03-31T15:56:26.025774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041438+0000","last_change":"2024-03-31T15:56:26.036552+0000","last_active":"2024-03-31T15:56:26.041438+0000","last_peered":"2024-03-31T15:56:26.041438+0000","last_clean":"2024-03-31T15:56:26.041438+0000","last_became_active":"2024-03-31T15:56:26.036319+0000","last_became_peered":"2024-03-31T15:56:26.036319+0000","last_unstale":"2024-03-31T15:56:26.041438+0000","last_undegraded":"2024-03-31T15:56:26.041438+0000","last_fullsized":"2024-03-31T15:56:26.041438+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021890+0000","last_change":"2024-03-31T15:56:26.021890+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021890+0000","last_undegraded":"2024-03-31T15:56:26.021890+0000","last_fullsized":"2024-03-31T15:56:26.021890+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042115+0000","last_change":"2024-03-31T15:56:26.039110+0000","last_active":"2024-03-31T15:56:26.042115+0000","last_peered":"2024-03-31T15:56:26.042115+0000","last_clean":"2024-03-31T15:56:26.042115+0000","last_became_active":"2024-03-31T15:56:26.037645+0000","last_became_peered":"2024-03-31T15:56:26.037645+0000","last_unstale":"2024-03-31T15:56:26.042115+0000","last_undegraded":"2024-03-31T15:56:26.042115+0000","last_fullsized":"2024-03-31T15:56:26.042115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:37:11.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023537+0000","last_change":"2024-03-31T15:56:26.023537+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023537+0000","last_undegraded":"2024-03-31T15:56:26.023537+0000","last_fullsized":"2024-03-31T15:56:26.023537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048112+0000","last_change":"2024-03-31T15:56:26.040713+0000","last_active":"2024-03-31T15:56:26.048112+0000","last_peered":"2024-03-31T15:56:26.048112+0000","last_clean":"2024-03-31T15:56:26.048112+0000","last_became_active":"2024-03-31T15:56:26.040391+0000","last_became_peered":"2024-03-31T15:56:26.040391+0000","last_unstale":"2024-03-31T15:56:26.048112+0000","last_undegraded":"2024-03-31T15:56:26.048112+0000","last_fullsized":"2024-03-31T15:56:26.048112+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027172+0000","last_change":"2024-03-31T15:56:26.027172+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027172+0000","last_undegraded":"2024-03-31T15:56:26.027172+0000","last_fullsized":"2024-03-31T15:56:26.027172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041125+0000","last_change":"2024-03-31T15:56:26.035712+0000","last_active":"2024-03-31T15:56:26.041125+0000","last_peered":"2024-03-31T15:56:26.041125+0000","last_clean":"2024-03-31T15:56:26.041125+0000","last_became_active":"2024-03-31T15:56:26.034040+0000","last_became_peered":"2024-03-31T15:56:26.034040+0000","last_unstale":"2024-03-31T15:56:26.041125+0000","last_undegraded":"2024-03-31T15:56:26.041125+0000","last_fullsized":"2024-03-31T15:56:26.041125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.039713+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.037623+0000","last_became_peered":"2024-03-31T15:56:27.037623+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:10.759953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041511+0000","last_change":"2024-03-31T15:56:26.036984+0000","last_active":"2024-03-31T15:56:26.041511+0000","last_peered":"2024-03-31T15:56:26.041511+0000","last_clean":"2024-03-31T15:56:26.041511+0000","last_became_active":"2024-03-31T15:56:26.036674+0000","last_became_peered":"2024-03-31T15:56:26.036674+0000","last_unstale":"2024-03-31T15:56:26.041511+0000","last_undegraded":"2024-03-31T15:56:26.041511+0000","last_fullsized":"2024-03-31T15:56:26.041511+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048608+0000","last_change":"2024-03-31T15:56:27.037261+0000","last_active":"2024-03-31T15:56:28.048608+0000","last_peered":"2024-03-31T15:56:28.048608+0000","last_clean":"2024-03-31T15:56:28.048608+0000","last_became_active":"2024-03-31T15:56:27.036282+0000","last_became_peered":"2024-03-31T15:56:27.036282+0000","last_unstale":"2024-03-31T15:56:28.048608+0000","last_undegraded":"2024-03-31T15:56:28.048608+0000","last_fullsized":"2024-03-31T15:56:28.048608+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:25.749230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.047955+0000","last_change":"2024-03-31T15:56:26.042322+0000","last_active":"2024-03-31T15:56:26.047955+0000","last_peered":"2024-03-31T15:56:26.047955+0000","last_clean":"2024-03-31T15:56:26.047955+0000","last_became_active":"2024-03-31T15:56:26.041942+0000","last_became_peered":"2024-03-31T15:56:26.041942+0000","last_unstale":"2024-03-31T15:56:26.047955+0000","last_undegraded":"2024-03-31T15:56:26.047955+0000","last_fullsized":"2024-03-31T15:56:26.047955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022096+0000","last_change":"2024-03-31T15:56:26.022096+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022096+0000","last_undegraded":"2024-03-31T15:56:26.022096+0000","last_fullsized":"2024-03-31T15:56:26.022096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.043837+0000","last_change":"2024-03-31T15:56:26.041462+0000","last_active":"2024-03-31T15:56:26.043837+0000","last_peered":"2024-03-31T15:56:26.043837+0000","last_clean":"2024-03-31T15:56:26.043837+0000","last_became_active":"2024-03-31T15:56:26.041184+0000","last_became_peered":"2024-03-31T15:56:26.041184+0000","last_unstale":"2024-03-31T15:56:26.043837+0000","last_undegraded":"2024-03-31T15:56:26.043837+0000","last_fullsized":"2024-03-31T15:56:26.043837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.042943+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.042359+0000","last_became_peered":"2024-03-31T15:56:27.042359+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:00.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733822+0000","last_change":"2024-03-31T15:56:26.036968+0000","last_active":"2024-03-31T15:56:28.733822+0000","last_peered":"2024-03-31T15:56:28.733822+0000","last_clean":"2024-03-31T15:56:28.733822+0000","last_became_active":"2024-03-31T15:56:26.036405+0000","last_became_peered":"2024-03-31T15:56:26.036405+0000","last_unstale":"2024-03-31T15:56:28.733822+0000","last_undegraded":"2024-03-31T15:56:28.733822+0000","last_fullsized":"2024-03-31T15:56:28.733822+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:28:07.298713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733795+0000","last_change":"2024-03-31T15:56:27.057654+0000","last_active":"2024-03-31T15:56:28.733795+0000","last_peered":"2024-03-31T15:56:28.733795+0000","last_clean":"2024-03-31T15:56:28.733795+0000","last_became_active":"2024-03-31T15:56:27.057124+0000","last_became_peered":"2024-03-31T15:56:27.057124+0000","last_unstale":"2024-03-31T15:56:28.733795+0000","last_undegraded":"2024-03-31T15:56:28.733795+0000","last_fullsized":"2024-03-31T15:56:28.733795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045156+0000","last_change":"2024-03-31T15:56:26.039648+0000","last_active":"2024-03-31T15:56:26.045156+0000","last_peered":"2024-03-31T15:56:26.045156+0000","last_clean":"2024-03-31T15:56:26.045156+0000","last_became_active":"2024-03-31T15:56:26.039276+0000","last_became_peered":"2024-03-31T15:56:26.039276+0000","last_unstale":"2024-03-31T15:56:26.045156+0000","last_undegraded":"2024-03-31T15:56:26.045156+0000","last_fullsized":"2024-03-31T15:56:26.045156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047349+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:28.047349+0000","last_peered":"2024-03-31T15:56:28.047349+0000","last_clean":"2024-03-31T15:56:28.047349+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:28.047349+0000","last_undegraded":"2024-03-31T15:56:28.047349+0000","last_fullsized":"2024-03-31T15:56:28.047349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":273822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":870849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1253376,"data_stored":884868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:56:31.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:56:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 Processor -- start 2024-03-31T15:56:31.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 -- start start 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7300d48d0 0x7ff73006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff73006c7c0 0x7ff7300cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff73015e8a0 con 0x7ff73015e4c0 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff73006cf90 con 0x7ff7300d48d0 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff738000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff73006d270 con 0x7ff73006c7c0 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff735600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff73015e4c0 0x7ff7300d41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50458/0 (socket says 172.21.15.90:50458) 2024-03-31T15:56:31.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff735600640 1 -- 172.21.15.90:0/1552904139 learned_addr learned my addr 172.21.15.90:0/1552904139 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff734c00640 1 --2- 172.21.15.90:0/1552904139 >> v2:172.21.15.102:3300/0 conn(0x7ff7300d48d0 0x7ff73006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.796+0000 7ff736000640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff73006c7c0 0x7ff7300cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff72fe00640 1 -- 172.21.15.90:0/1552904139 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831886853 0 0) 0x7ff73015e8a0 con 0x7ff73015e4c0 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff72fe00640 1 -- 172.21.15.90:0/1552904139 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff718003530 con 0x7ff73015e4c0 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff734c00640 1 -- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff73006c7c0 msgr2=0x7ff7300cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff734c00640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff73006c7c0 0x7ff7300cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff734c00640 1 -- 172.21.15.90:0/1552904139 >> v1:172.21.15.90:6789/0 conn(0x7ff73015e4c0 legacy=0x7ff7300d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff734c00640 1 -- 172.21.15.90:0/1552904139 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73006d550 con 0x7ff7300d48d0 2024-03-31T15:56:31.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff734c00640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7300d48d0 0x7ff73006c280 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0097c0 tx=0x7ff71c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97a045bad806052 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff72fe00640 1 -- 172.21.15.90:0/1552904139 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0aa5f0 con 0x7ff7300d48d0 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff72fe00640 1 -- 172.21.15.90:0/1552904139 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff71c0af070 con 0x7ff7300d48d0 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.800+0000 7ff72fe00640 1 -- 172.21.15.90:0/1552904139 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0aa9b0 con 0x7ff7300d48d0 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7300d48d0 msgr2=0x7ff73006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7300d48d0 0x7ff73006c280 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0097c0 tx=0x7ff71c01c470 comp rx=0 tx=0).stop 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/1552904139 shutdown_connections 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff73006c7c0 0x7ff7300cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- 172.21.15.90:0/1552904139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7300d48d0 0x7ff73006c280 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/1552904139 >> 172.21.15.90:0/1552904139 conn(0x7ff73008db60 msgr2=0x7ff73009d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/1552904139 shutdown_connections 2024-03-31T15:56:31.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/1552904139 wait complete. 2024-03-31T15:56:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 Processor -- start 2024-03-31T15:56:31.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- start start 2024-03-31T15:56:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7300d48d0 0x7ff73015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff73015e4c0 0x7ff73014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7300d0350 con 0x7ff73006c7c0 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7300d01d0 con 0x7ff73015e4c0 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff7300d04d0 con 0x7ff7300d48d0 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41942/0 (socket says 172.21.15.90:41942) 2024-03-31T15:56:31.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 -- 172.21.15.90:0/2142884628 learned_addr learned my addr 172.21.15.90:0/2142884628 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 -- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7300d48d0 msgr2=0x7ff73015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7300d48d0 0x7ff73015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 -- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff73015e4c0 msgr2=0x7ff73014de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff73015e4c0 0x7ff73014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 -- 172.21.15.90:0/2142884628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73015b5a0 con 0x7ff73006c7c0 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff735600640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff7240010f0 tx=0x7ff72400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724017070 con 0x7ff73006c7c0 2024-03-31T15:56:31.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff724005d10 con 0x7ff73006c7c0 2024-03-31T15:56:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724007310 con 0x7ff73006c7c0 2024-03-31T15:56:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.804+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff730151430 con 0x7ff73006c7c0 2024-03-31T15:56:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73015bb40 con 0x7ff73006c7c0 2024-03-31T15:56:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7300686d0 con 0x7ff73006c7c0 2024-03-31T15:56:31.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff72401a070 con 0x7ff73006c7c0 2024-03-31T15:56:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff72d600640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 0x7ff71006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7ff724085ef0 con 0x7ff73006c7c0 2024-03-31T15:56:31.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff734c00640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 0x7ff71006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:31.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.808+0000 7ff734c00640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 0x7ff71006a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ae000 tx=0x7ff71c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:31.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.812+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff724056310 con 0x7ff73006c7c0 2024-03-31T15:56:31.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.968+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff73015c880 con 0x7ff710068380 2024-03-31T15:56:31.968 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff72d600640 1 -- 172.21.15.90:0/2142884628 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249206 (secure 0 0 0) 0x7ff73015c880 con 0x7ff710068380 2024-03-31T15:56:31.970 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:31.972 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 msgr2=0x7ff71006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 0x7ff71006a840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff71c0ae000 tx=0x7ff71c004740 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 msgr2=0x7ff7300cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff7240010f0 tx=0x7ff72400c890 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 shutdown_connections 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff710068380 0x7ff71006a840 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff73015e4c0 0x7ff73014de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7300d48d0 0x7ff73015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 --2- 172.21.15.90:0/2142884628 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff73006c7c0 0x7ff7300cb300 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:31.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 >> 172.21.15.90:0/2142884628 conn(0x7ff73008db60 msgr2=0x7ff7300d1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:31.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 shutdown_connections 2024-03-31T15:56:31.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:31.976+0000 7ff738000640 1 -- 172.21.15.90:0/2142884628 wait complete. 2024-03-31T15:56:31.990 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-31T15:56:31.255055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590514,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":180,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222312,"kb_used_data":6140,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752408,"statfs":{"total":773094113280,"available":772866465792,"internally_reserved":0,"allocated":6287360,"data_stored":3459774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001246"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732549+0000","last_change":"2024-03-31T15:56:27.060874+0000","last_active":"2024-03-31T15:56:28.732549+0000","last_peered":"2024-03-31T15:56:28.732549+0000","last_clean":"2024-03-31T15:56:28.732549+0000","last_became_active":"2024-03-31T15:56:27.060630+0000","last_became_peered":"2024-03-31T15:56:27.060630+0000","last_unstale":"2024-03-31T15:56:28.732549+0000","last_undegraded":"2024-03-31T15:56:28.732549+0000","last_fullsized":"2024-03-31T15:56:28.732549+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:56:54.783818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734232+0000","last_change":"2024-03-31T15:56:27.040029+0000","last_active":"2024-03-31T15:56:28.734232+0000","last_peered":"2024-03-31T15:56:28.734232+0000","last_clean":"2024-03-31T15:56:28.734232+0000","last_became_active":"2024-03-31T15:56:27.036544+0000","last_became_peered":"2024-03-31T15:56:27.036544+0000","last_unstale":"2024-03-31T15:56:28.734232+0000","last_undegraded":"2024-03-31T15:56:28.734232+0000","last_fullsized":"2024-03-31T15:56:28.734232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:57.935165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048573+0000","last_change":"2024-03-31T15:56:27.040254+0000","last_active":"2024-03-31T15:56:28.048573+0000","last_peered":"2024-03-31T15:56:28.048573+0000","last_clean":"2024-03-31T15:56:28.048573+0000","last_became_active":"2024-03-31T15:56:27.039839+0000","last_became_peered":"2024-03-31T15:56:27.039839+0000","last_unstale":"2024-03-31T15:56:28.048573+0000","last_undegraded":"2024-03-31T15:56:28.048573+0000","last_fullsized":"2024-03-31T15:56:28.048573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:38.034420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021551+0000","last_change":"2024-03-31T15:56:26.021551+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021551+0000","last_undegraded":"2024-03-31T15:56:26.021551+0000","last_fullsized":"2024-03-31T15:56:26.021551+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731537+0000","last_change":"2024-03-31T15:56:27.057953+0000","last_active":"2024-03-31T15:56:28.731537+0000","last_peered":"2024-03-31T15:56:28.731537+0000","last_clean":"2024-03-31T15:56:28.731537+0000","last_became_active":"2024-03-31T15:56:27.057680+0000","last_became_peered":"2024-03-31T15:56:27.057680+0000","last_unstale":"2024-03-31T15:56:28.731537+0000","last_undegraded":"2024-03-31T15:56:28.731537+0000","last_fullsized":"2024-03-31T15:56:28.731537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:41:20.883103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025926+0000","last_change":"2024-03-31T15:56:26.025926+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025926+0000","last_undegraded":"2024-03-31T15:56:26.025926+0000","last_fullsized":"2024-03-31T15:56:26.025926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028615+0000","last_change":"2024-03-31T15:56:26.028615+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028615+0000","last_undegraded":"2024-03-31T15:56:26.028615+0000","last_fullsized":"2024-03-31T15:56:26.028615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733707+0000","last_change":"2024-03-31T15:56:27.040713+0000","last_active":"2024-03-31T15:56:28.733707+0000","last_peered":"2024-03-31T15:56:28.733707+0000","last_clean":"2024-03-31T15:56:28.733707+0000","last_became_active":"2024-03-31T15:56:27.040439+0000","last_became_peered":"2024-03-31T15:56:27.040439+0000","last_unstale":"2024-03-31T15:56:28.733707+0000","last_undegraded":"2024-03-31T15:56:28.733707+0000","last_fullsized":"2024-03-31T15:56:28.733707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026552+0000","last_change":"2024-03-31T15:56:26.026552+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026552+0000","last_undegraded":"2024-03-31T15:56:26.026552+0000","last_fullsized":"2024-03-31T15:56:26.026552+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022133+0000","last_change":"2024-03-31T15:56:26.022133+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022133+0000","last_undegraded":"2024-03-31T15:56:26.022133+0000","last_fullsized":"2024-03-31T15:56:26.022133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022163+0000","last_change":"2024-03-31T15:56:26.022163+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022163+0000","last_undegraded":"2024-03-31T15:56:26.022163+0000","last_fullsized":"2024-03-31T15:56:26.022163+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021075+0000","last_change":"2024-03-31T15:56:26.021075+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021075+0000","last_undegraded":"2024-03-31T15:56:26.021075+0000","last_fullsized":"2024-03-31T15:56:26.021075+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028321+0000","last_change":"2024-03-31T15:56:26.028321+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028321+0000","last_undegraded":"2024-03-31T15:56:26.028321+0000","last_fullsized":"2024-03-31T15:56:26.028321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023949+0000","last_change":"2024-03-31T15:56:26.023949+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023949+0000","last_undegraded":"2024-03-31T15:56:26.023949+0000","last_fullsized":"2024-03-31T15:56:26.023949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027463+0000","last_change":"2024-03-31T15:56:26.027463+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027463+0000","last_undegraded":"2024-03-31T15:56:26.027463+0000","last_fullsized":"2024-03-31T15:56:26.027463+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025451+0000","last_change":"2024-03-31T15:56:26.025451+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025451+0000","last_undegraded":"2024-03-31T15:56:26.025451+0000","last_fullsized":"2024-03-31T15:56:26.025451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734398+0000","last_change":"2024-03-31T15:56:27.041513+0000","last_active":"2024-03-31T15:56:28.734398+0000","last_peered":"2024-03-31T15:56:28.734398+0000","last_clean":"2024-03-31T15:56:28.734398+0000","last_became_active":"2024-03-31T15:56:27.041210+0000","last_became_peered":"2024-03-31T15:56:27.041210+0000","last_unstale":"2024-03-31T15:56:28.734398+0000","last_undegraded":"2024-03-31T15:56:28.734398+0000","last_fullsized":"2024-03-31T15:56:28.734398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:55.146299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048591+0000","last_change":"2024-03-31T15:56:27.049282+0000","last_active":"2024-03-31T15:56:28.048591+0000","last_peered":"2024-03-31T15:56:28.048591+0000","last_clean":"2024-03-31T15:56:28.048591+0000","last_became_active":"2024-03-31T15:56:27.048536+0000","last_became_peered":"2024-03-31T15:56:27.048536+0000","last_unstale":"2024-03-31T15:56:28.048591+0000","last_undegraded":"2024-03-31T15:56:28.048591+0000","last_fullsized":"2024-03-31T15:56:28.048591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:49:37.697269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732485+0000","last_change":"2024-03-31T15:56:27.060292+0000","last_active":"2024-03-31T15:56:28.732485+0000","last_peered":"2024-03-31T15:56:28.732485+0000","last_clean":"2024-03-31T15:56:28.732485+0000","last_became_active":"2024-03-31T15:56:27.060044+0000","last_became_peered":"2024-03-31T15:56:27.060044+0000","last_unstale":"2024-03-31T15:56:28.732485+0000","last_undegraded":"2024-03-31T15:56:28.732485+0000","last_fullsized":"2024-03-31T15:56:28.732485+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:03.617564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734311+0000","last_change":"2024-03-31T15:56:27.059752+0000","last_active":"2024-03-31T15:56:28.734311+0000","last_peered":"2024-03-31T15:56:28.734311+0000","last_clean":"2024-03-31T15:56:28.734311+0000","last_became_active":"2024-03-31T15:56:27.059504+0000","last_became_peered":"2024-03-31T15:56:27.059504+0000","last_unstale":"2024-03-31T15:56:28.734311+0000","last_undegraded":"2024-03-31T15:56:28.734311+0000","last_fullsized":"2024-03-31T15:56:28.734311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:03.532453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048464+0000","last_change":"2024-03-31T15:56:27.030882+0000","last_active":"2024-03-31T15:56:28.048464+0000","last_peered":"2024-03-31T15:56:28.048464+0000","last_clean":"2024-03-31T15:56:28.048464+0000","last_became_active":"2024-03-31T15:56:27.030222+0000","last_became_peered":"2024-03-31T15:56:27.030222+0000","last_unstale":"2024-03-31T15:56:28.048464+0000","last_undegraded":"2024-03-31T15:56:28.048464+0000","last_fullsized":"2024-03-31T15:56:28.048464+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:25.459268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733876+0000","last_change":"2024-03-31T15:56:27.053182+0000","last_active":"2024-03-31T15:56:28.733876+0000","last_peered":"2024-03-31T15:56:28.733876+0000","last_clean":"2024-03-31T15:56:28.733876+0000","last_became_active":"2024-03-31T15:56:27.052882+0000","last_became_peered":"2024-03-31T15:56:27.052882+0000","last_unstale":"2024-03-31T15:56:28.733876+0000","last_undegraded":"2024-03-31T15:56:28.733876+0000","last_fullsized":"2024-03-31T15:56:28.733876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047955+0000","last_change":"2024-03-31T15:56:27.037326+0000","last_active":"2024-03-31T15:56:28.047955+0000","last_peered":"2024-03-31T15:56:28.047955+0000","last_clean":"2024-03-31T15:56:28.047955+0000","last_became_active":"2024-03-31T15:56:27.036842+0000","last_became_peered":"2024-03-31T15:56:27.036842+0000","last_unstale":"2024-03-31T15:56:28.047955+0000","last_undegraded":"2024-03-31T15:56:28.047955+0000","last_fullsized":"2024-03-31T15:56:28.047955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:13.092644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026315+0000","last_change":"2024-03-31T15:56:26.026315+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026315+0000","last_undegraded":"2024-03-31T15:56:26.026315+0000","last_fullsized":"2024-03-31T15:56:26.026315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732648+0000","last_change":"2024-03-31T15:56:27.058506+0000","last_active":"2024-03-31T15:56:28.732648+0000","last_peered":"2024-03-31T15:56:28.732648+0000","last_clean":"2024-03-31T15:56:28.732648+0000","last_became_active":"2024-03-31T15:56:27.058252+0000","last_became_peered":"2024-03-31T15:56:27.058252+0000","last_unstale":"2024-03-31T15:56:28.732648+0000","last_undegraded":"2024-03-31T15:56:28.732648+0000","last_fullsized":"2024-03-31T15:56:28.732648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:03.064057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026205+0000","last_change":"2024-03-31T15:56:26.026205+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026205+0000","last_undegraded":"2024-03-31T15:56:26.026205+0000","last_fullsized":"2024-03-31T15:56:26.026205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031242+0000","last_change":"2024-03-31T15:56:26.031242+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031242+0000","last_undegraded":"2024-03-31T15:56:26.031242+0000","last_fullsized":"2024-03-31T15:56:26.031242+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732586+0000","last_change":"2024-03-31T15:56:27.058392+0000","last_active":"2024-03-31T15:56:28.732586+0000","last_peered":"2024-03-31T15:56:28.732586+0000","last_clean":"2024-03-31T15:56:28.732586+0000","last_became_active":"2024-03-31T15:56:27.058103+0000","last_became_peered":"2024-03-31T15:56:27.058103+0000","last_unstale":"2024-03-31T15:56:28.732586+0000","last_undegraded":"2024-03-31T15:56:28.732586+0000","last_fullsized":"2024-03-31T15:56:28.732586+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:38.875392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734201+0000","last_change":"2024-03-31T15:56:27.041576+0000","last_active":"2024-03-31T15:56:28.734201+0000","last_peered":"2024-03-31T15:56:28.734201+0000","last_clean":"2024-03-31T15:56:28.734201+0000","last_became_active":"2024-03-31T15:56:27.041327+0000","last_became_peered":"2024-03-31T15:56:27.041327+0000","last_unstale":"2024-03-31T15:56:28.734201+0000","last_undegraded":"2024-03-31T15:56:28.734201+0000","last_fullsized":"2024-03-31T15:56:28.734201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:39.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048359+0000","last_change":"2024-03-31T15:56:27.034473+0000","last_active":"2024-03-31T15:56:28.048359+0000","last_peered":"2024-03-31T15:56:28.048359+0000","last_clean":"2024-03-31T15:56:28.048359+0000","last_became_active":"2024-03-31T15:56:27.034209+0000","last_became_peered":"2024-03-31T15:56:27.034209+0000","last_unstale":"2024-03-31T15:56:28.048359+0000","last_undegraded":"2024-03-31T15:56:28.048359+0000","last_fullsized":"2024-03-31T15:56:28.048359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:45.169835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731505+0000","last_change":"2024-03-31T15:56:27.058056+0000","last_active":"2024-03-31T15:56:28.731505+0000","last_peered":"2024-03-31T15:56:28.731505+0000","last_clean":"2024-03-31T15:56:28.731505+0000","last_became_active":"2024-03-31T15:56:27.057786+0000","last_became_peered":"2024-03-31T15:56:27.057786+0000","last_unstale":"2024-03-31T15:56:28.731505+0000","last_undegraded":"2024-03-31T15:56:28.731505+0000","last_fullsized":"2024-03-31T15:56:28.731505+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:04:01.295156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024954+0000","last_change":"2024-03-31T15:56:26.024954+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024954+0000","last_undegraded":"2024-03-31T15:56:26.024954+0000","last_fullsized":"2024-03-31T15:56:26.024954+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031116+0000","last_change":"2024-03-31T15:56:26.031116+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031116+0000","last_undegraded":"2024-03-31T15:56:26.031116+0000","last_fullsized":"2024-03-31T15:56:26.031116+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044934+0000","last_change":"2024-03-31T15:56:26.043587+0000","last_active":"2024-03-31T15:56:26.044934+0000","last_peered":"2024-03-31T15:56:26.044934+0000","last_clean":"2024-03-31T15:56:26.044934+0000","last_became_active":"2024-03-31T15:56:26.043321+0000","last_became_peered":"2024-03-31T15:56:26.043321+0000","last_unstale":"2024-03-31T15:56:26.044934+0000","last_undegraded":"2024-03-31T15:56:26.044934+0000","last_fullsized":"2024-03-31T15:56:26.044934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027162+0000","last_change":"2024-03-31T15:56:26.027162+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027162+0000","last_undegraded":"2024-03-31T15:56:26.027162+0000","last_fullsized":"2024-03-31T15:56:26.027162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734183+0000","last_change":"2024-03-31T15:56:26.029688+0000","last_active":"2024-03-31T15:56:28.734183+0000","last_peered":"2024-03-31T15:56:28.734183+0000","last_clean":"2024-03-31T15:56:28.734183+0000","last_became_active":"2024-03-31T15:56:26.029396+0000","last_became_peered":"2024-03-31T15:56:26.029396+0000","last_unstale":"2024-03-31T15:56:28.734183+0000","last_undegraded":"2024-03-31T15:56:28.734183+0000","last_fullsized":"2024-03-31T15:56:28.734183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:52:46.291141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026382+0000","last_change":"2024-03-31T15:56:26.026382+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026382+0000","last_undegraded":"2024-03-31T15:56:26.026382+0000","last_fullsized":"2024-03-31T15:56:26.026382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045804+0000","last_change":"2024-03-31T15:56:26.044595+0000","last_active":"2024-03-31T15:56:26.045804+0000","last_peered":"2024-03-31T15:56:26.045804+0000","last_clean":"2024-03-31T15:56:26.045804+0000","last_became_active":"2024-03-31T15:56:26.044321+0000","last_became_peered":"2024-03-31T15:56:26.044321+0000","last_unstale":"2024-03-31T15:56:26.045804+0000","last_undegraded":"2024-03-31T15:56:26.045804+0000","last_fullsized":"2024-03-31T15:56:26.045804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027580+0000","last_change":"2024-03-31T15:56:26.027580+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027580+0000","last_undegraded":"2024-03-31T15:56:26.027580+0000","last_fullsized":"2024-03-31T15:56:26.027580+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.038619+0000","last_change":"2024-03-31T15:56:26.035102+0000","last_active":"2024-03-31T15:56:26.038619+0000","last_peered":"2024-03-31T15:56:26.038619+0000","last_clean":"2024-03-31T15:56:26.038619+0000","last_became_active":"2024-03-31T15:56:26.034821+0000","last_became_peered":"2024-03-31T15:56:26.034821+0000","last_unstale":"2024-03-31T15:56:26.038619+0000","last_undegraded":"2024-03-31T15:56:26.038619+0000","last_fullsized":"2024-03-31T15:56:26.038619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733891+0000","last_change":"2024-03-31T15:56:27.053152+0000","last_active":"2024-03-31T15:56:28.733891+0000","last_peered":"2024-03-31T15:56:28.733891+0000","last_clean":"2024-03-31T15:56:28.733891+0000","last_became_active":"2024-03-31T15:56:27.052870+0000","last_became_peered":"2024-03-31T15:56:27.052870+0000","last_unstale":"2024-03-31T15:56:28.733891+0000","last_undegraded":"2024-03-31T15:56:28.733891+0000","last_fullsized":"2024-03-31T15:56:28.733891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:53.273112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042800+0000","last_change":"2024-03-31T15:56:26.035499+0000","last_active":"2024-03-31T15:56:26.042800+0000","last_peered":"2024-03-31T15:56:26.042800+0000","last_clean":"2024-03-31T15:56:26.042800+0000","last_became_active":"2024-03-31T15:56:26.035216+0000","last_became_peered":"2024-03-31T15:56:26.035216+0000","last_unstale":"2024-03-31T15:56:26.042800+0000","last_undegraded":"2024-03-31T15:56:26.042800+0000","last_fullsized":"2024-03-31T15:56:26.042800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734108+0000","last_change":"2024-03-31T15:56:27.055018+0000","last_active":"2024-03-31T15:56:28.734108+0000","last_peered":"2024-03-31T15:56:28.734108+0000","last_clean":"2024-03-31T15:56:28.734108+0000","last_became_active":"2024-03-31T15:56:27.054765+0000","last_became_peered":"2024-03-31T15:56:27.054765+0000","last_unstale":"2024-03-31T15:56:28.734108+0000","last_undegraded":"2024-03-31T15:56:28.734108+0000","last_fullsized":"2024-03-31T15:56:28.734108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:49.516522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044103+0000","last_change":"2024-03-31T15:56:26.040614+0000","last_active":"2024-03-31T15:56:26.044103+0000","last_peered":"2024-03-31T15:56:26.044103+0000","last_clean":"2024-03-31T15:56:26.044103+0000","last_became_active":"2024-03-31T15:56:26.040371+0000","last_became_peered":"2024-03-31T15:56:26.040371+0000","last_unstale":"2024-03-31T15:56:26.044103+0000","last_undegraded":"2024-03-31T15:56:26.044103+0000","last_fullsized":"2024-03-31T15:56:26.044103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025656+0000","last_change":"2024-03-31T15:56:26.025656+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025656+0000","last_undegraded":"2024-03-31T15:56:26.025656+0000","last_fullsized":"2024-03-31T15:56:26.025656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041171+0000","last_change":"2024-03-31T15:56:26.037304+0000","last_active":"2024-03-31T15:56:26.041171+0000","last_peered":"2024-03-31T15:56:26.041171+0000","last_clean":"2024-03-31T15:56:26.041171+0000","last_became_active":"2024-03-31T15:56:26.037038+0000","last_became_peered":"2024-03-31T15:56:26.037038+0000","last_unstale":"2024-03-31T15:56:26.041171+0000","last_undegraded":"2024-03-31T15:56:26.041171+0000","last_fullsized":"2024-03-31T15:56:26.041171+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:52.264596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733928+0000","last_change":"2024-03-31T15:56:27.043180+0000","last_active":"2024-03-31T15:56:28.733928+0000","last_peered":"2024-03-31T15:56:28.733928+0000","last_clean":"2024-03-31T15:56:28.733928+0000","last_became_active":"2024-03-31T15:56:27.042692+0000","last_became_peered":"2024-03-31T15:56:27.042692+0000","last_unstale":"2024-03-31T15:56:28.733928+0000","last_undegraded":"2024-03-31T15:56:28.733928+0000","last_fullsized":"2024-03-31T15:56:28.733928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:21:35.971485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045526+0000","last_change":"2024-03-31T15:56:26.040038+0000","last_active":"2024-03-31T15:56:26.045526+0000","last_peered":"2024-03-31T15:56:26.045526+0000","last_clean":"2024-03-31T15:56:26.045526+0000","last_became_active":"2024-03-31T15:56:26.039654+0000","last_became_peered":"2024-03-31T15:56:26.039654+0000","last_unstale":"2024-03-31T15:56:26.045526+0000","last_undegraded":"2024-03-31T15:56:26.045526+0000","last_fullsized":"2024-03-31T15:56:26.045526+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023782+0000","last_change":"2024-03-31T15:56:26.023782+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023782+0000","last_undegraded":"2024-03-31T15:56:26.023782+0000","last_fullsized":"2024-03-31T15:56:26.023782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049029+0000","last_change":"2024-03-31T15:56:26.041995+0000","last_active":"2024-03-31T15:56:28.049029+0000","last_peered":"2024-03-31T15:56:28.049029+0000","last_clean":"2024-03-31T15:56:28.049029+0000","last_became_active":"2024-03-31T15:56:26.041698+0000","last_became_peered":"2024-03-31T15:56:26.041698+0000","last_unstale":"2024-03-31T15:56:28.049029+0000","last_undegraded":"2024-03-31T15:56:28.049029+0000","last_fullsized":"2024-03-31T15:56:28.049029+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:03.382018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049076+0000","last_change":"2024-03-31T15:56:27.032991+0000","last_active":"2024-03-31T15:56:28.049076+0000","last_peered":"2024-03-31T15:56:28.049076+0000","last_clean":"2024-03-31T15:56:28.049076+0000","last_became_active":"2024-03-31T15:56:27.032057+0000","last_became_peered":"2024-03-31T15:56:27.032057+0000","last_unstale":"2024-03-31T15:56:28.049076+0000","last_undegraded":"2024-03-31T15:56:28.049076+0000","last_fullsized":"2024-03-31T15:56:28.049076+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734295+0000","last_change":"2024-03-31T15:56:26.034420+0000","last_active":"2024-03-31T15:56:28.734295+0000","last_peered":"2024-03-31T15:56:28.734295+0000","last_clean":"2024-03-31T15:56:28.734295+0000","last_became_active":"2024-03-31T15:56:26.034145+0000","last_became_peered":"2024-03-31T15:56:26.034145+0000","last_unstale":"2024-03-31T15:56:28.734295+0000","last_undegraded":"2024-03-31T15:56:28.734295+0000","last_fullsized":"2024-03-31T15:56:28.734295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:18.044685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023933+0000","last_change":"2024-03-31T15:56:26.023933+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023933+0000","last_undegraded":"2024-03-31T15:56:26.023933+0000","last_fullsized":"2024-03-31T15:56:26.023933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048878+0000","last_change":"2024-03-31T15:56:26.044785+0000","last_active":"2024-03-31T15:56:28.048878+0000","last_peered":"2024-03-31T15:56:28.048878+0000","last_clean":"2024-03-31T15:56:28.048878+0000","last_became_active":"2024-03-31T15:56:26.044515+0000","last_became_peered":"2024-03-31T15:56:26.044515+0000","last_unstale":"2024-03-31T15:56:28.048878+0000","last_undegraded":"2024-03-31T15:56:28.048878+0000","last_fullsized":"2024-03-31T15:56:28.048878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:17.779102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048748+0000","last_change":"2024-03-31T15:56:27.032521+0000","last_active":"2024-03-31T15:56:28.048748+0000","last_peered":"2024-03-31T15:56:28.048748+0000","last_clean":"2024-03-31T15:56:28.048748+0000","last_became_active":"2024-03-31T15:56:27.032235+0000","last_became_peered":"2024-03-31T15:56:27.032235+0000","last_unstale":"2024-03-31T15:56:28.048748+0000","last_undegraded":"2024-03-31T15:56:28.048748+0000","last_fullsized":"2024-03-31T15:56:28.048748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045554+0000","last_change":"2024-03-31T15:56:26.041173+0000","last_active":"2024-03-31T15:56:26.045554+0000","last_peered":"2024-03-31T15:56:26.045554+0000","last_clean":"2024-03-31T15:56:26.045554+0000","last_became_active":"2024-03-31T15:56:26.040901+0000","last_became_peered":"2024-03-31T15:56:26.040901+0000","last_unstale":"2024-03-31T15:56:26.045554+0000","last_undegraded":"2024-03-31T15:56:26.045554+0000","last_fullsized":"2024-03-31T15:56:26.045554+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022785+0000","last_change":"2024-03-31T15:56:26.022785+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022785+0000","last_undegraded":"2024-03-31T15:56:26.022785+0000","last_fullsized":"2024-03-31T15:56:26.022785+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733993+0000","last_change":"2024-03-31T15:56:26.035713+0000","last_active":"2024-03-31T15:56:28.733993+0000","last_peered":"2024-03-31T15:56:28.733993+0000","last_clean":"2024-03-31T15:56:28.733993+0000","last_became_active":"2024-03-31T15:56:26.035393+0000","last_became_peered":"2024-03-31T15:56:26.035393+0000","last_unstale":"2024-03-31T15:56:28.733993+0000","last_undegraded":"2024-03-31T15:56:28.733993+0000","last_fullsized":"2024-03-31T15:56:28.733993+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:10.726676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048441+0000","last_change":"2024-03-31T15:56:27.041906+0000","last_active":"2024-03-31T15:56:28.048441+0000","last_peered":"2024-03-31T15:56:28.048441+0000","last_clean":"2024-03-31T15:56:28.048441+0000","last_became_active":"2024-03-31T15:56:27.041609+0000","last_became_peered":"2024-03-31T15:56:27.041609+0000","last_unstale":"2024-03-31T15:56:28.048441+0000","last_undegraded":"2024-03-31T15:56:28.048441+0000","last_fullsized":"2024-03-31T15:56:28.048441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:53:45.939898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048116+0000","last_change":"2024-03-31T15:56:26.041375+0000","last_active":"2024-03-31T15:56:26.048116+0000","last_peered":"2024-03-31T15:56:26.048116+0000","last_clean":"2024-03-31T15:56:26.048116+0000","last_became_active":"2024-03-31T15:56:26.041158+0000","last_became_peered":"2024-03-31T15:56:26.041158+0000","last_unstale":"2024-03-31T15:56:26.048116+0000","last_undegraded":"2024-03-31T15:56:26.048116+0000","last_fullsized":"2024-03-31T15:56:26.048116+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024021+0000","last_change":"2024-03-31T15:56:26.024021+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024021+0000","last_undegraded":"2024-03-31T15:56:26.024021+0000","last_fullsized":"2024-03-31T15:56:26.024021+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041150+0000","last_change":"2024-03-31T15:56:26.037434+0000","last_active":"2024-03-31T15:56:26.041150+0000","last_peered":"2024-03-31T15:56:26.041150+0000","last_clean":"2024-03-31T15:56:26.041150+0000","last_became_active":"2024-03-31T15:56:26.036753+0000","last_became_peered":"2024-03-31T15:56:26.036753+0000","last_unstale":"2024-03-31T15:56:26.041150+0000","last_undegraded":"2024-03-31T15:56:26.041150+0000","last_fullsized":"2024-03-31T15:56:26.041150+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.472784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047988+0000","last_change":"2024-03-31T15:56:27.037112+0000","last_active":"2024-03-31T15:56:28.047988+0000","last_peered":"2024-03-31T15:56:28.047988+0000","last_clean":"2024-03-31T15:56:28.047988+0000","last_became_active":"2024-03-31T15:56:27.036824+0000","last_became_peered":"2024-03-31T15:56:27.036824+0000","last_unstale":"2024-03-31T15:56:28.047988+0000","last_undegraded":"2024-03-31T15:56:28.047988+0000","last_fullsized":"2024-03-31T15:56:28.047988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:44:17.091367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047288+0000","last_change":"2024-03-31T15:56:26.038121+0000","last_active":"2024-03-31T15:56:28.047288+0000","last_peered":"2024-03-31T15:56:28.047288+0000","last_clean":"2024-03-31T15:56:28.047288+0000","last_became_active":"2024-03-31T15:56:26.037834+0000","last_became_peered":"2024-03-31T15:56:26.037834+0000","last_unstale":"2024-03-31T15:56:28.047288+0000","last_undegraded":"2024-03-31T15:56:28.047288+0000","last_fullsized":"2024-03-31T15:56:28.047288+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:38.551620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048667+0000","last_change":"2024-03-31T15:56:27.038762+0000","last_active":"2024-03-31T15:56:28.048667+0000","last_peered":"2024-03-31T15:56:28.048667+0000","last_clean":"2024-03-31T15:56:28.048667+0000","last_became_active":"2024-03-31T15:56:27.038442+0000","last_became_peered":"2024-03-31T15:56:27.038442+0000","last_unstale":"2024-03-31T15:56:28.048667+0000","last_undegraded":"2024-03-31T15:56:28.048667+0000","last_fullsized":"2024-03-31T15:56:28.048667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:23.516599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734348+0000","last_change":"2024-03-31T15:56:26.034027+0000","last_active":"2024-03-31T15:56:28.734348+0000","last_peered":"2024-03-31T15:56:28.734348+0000","last_clean":"2024-03-31T15:56:28.734348+0000","last_became_active":"2024-03-31T15:56:26.033735+0000","last_became_peered":"2024-03-31T15:56:26.033735+0000","last_unstale":"2024-03-31T15:56:28.734348+0000","last_undegraded":"2024-03-31T15:56:28.734348+0000","last_fullsized":"2024-03-31T15:56:28.734348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:46.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027856+0000","last_change":"2024-03-31T15:56:26.027856+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027856+0000","last_undegraded":"2024-03-31T15:56:26.027856+0000","last_fullsized":"2024-03-31T15:56:26.027856+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042183+0000","last_change":"2024-03-31T15:56:26.039370+0000","last_active":"2024-03-31T15:56:26.042183+0000","last_peered":"2024-03-31T15:56:26.042183+0000","last_clean":"2024-03-31T15:56:26.042183+0000","last_became_active":"2024-03-31T15:56:26.038769+0000","last_became_peered":"2024-03-31T15:56:26.038769+0000","last_unstale":"2024-03-31T15:56:26.042183+0000","last_undegraded":"2024-03-31T15:56:26.042183+0000","last_fullsized":"2024-03-31T15:56:26.042183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:36.882522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734424+0000","last_change":"2024-03-31T15:56:27.040609+0000","last_active":"2024-03-31T15:56:28.734424+0000","last_peered":"2024-03-31T15:56:28.734424+0000","last_clean":"2024-03-31T15:56:28.734424+0000","last_became_active":"2024-03-31T15:56:27.040329+0000","last_became_peered":"2024-03-31T15:56:27.040329+0000","last_unstale":"2024-03-31T15:56:28.734424+0000","last_undegraded":"2024-03-31T15:56:28.734424+0000","last_fullsized":"2024-03-31T15:56:28.734424+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:25.079313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041058+0000","last_change":"2024-03-31T15:56:26.036191+0000","last_active":"2024-03-31T15:56:26.041058+0000","last_peered":"2024-03-31T15:56:26.041058+0000","last_clean":"2024-03-31T15:56:26.041058+0000","last_became_active":"2024-03-31T15:56:26.035873+0000","last_became_peered":"2024-03-31T15:56:26.035873+0000","last_unstale":"2024-03-31T15:56:26.041058+0000","last_undegraded":"2024-03-31T15:56:26.041058+0000","last_fullsized":"2024-03-31T15:56:26.041058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048941+0000","last_change":"2024-03-31T15:56:27.034376+0000","last_active":"2024-03-31T15:56:28.048941+0000","last_peered":"2024-03-31T15:56:28.048941+0000","last_clean":"2024-03-31T15:56:28.048941+0000","last_became_active":"2024-03-31T15:56:27.032545+0000","last_became_peered":"2024-03-31T15:56:27.032545+0000","last_unstale":"2024-03-31T15:56:28.048941+0000","last_undegraded":"2024-03-31T15:56:28.048941+0000","last_fullsized":"2024-03-31T15:56:28.048941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045774+0000","last_change":"2024-03-31T15:56:26.044352+0000","last_active":"2024-03-31T15:56:26.045774+0000","last_peered":"2024-03-31T15:56:26.045774+0000","last_clean":"2024-03-31T15:56:26.045774+0000","last_became_active":"2024-03-31T15:56:26.044071+0000","last_became_peered":"2024-03-31T15:56:26.044071+0000","last_unstale":"2024-03-31T15:56:26.045774+0000","last_undegraded":"2024-03-31T15:56:26.045774+0000","last_fullsized":"2024-03-31T15:56:26.045774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022357+0000","last_change":"2024-03-31T15:56:26.022357+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022357+0000","last_undegraded":"2024-03-31T15:56:26.022357+0000","last_fullsized":"2024-03-31T15:56:26.022357+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734169+0000","last_change":"2024-03-31T15:56:26.030253+0000","last_active":"2024-03-31T15:56:28.734169+0000","last_peered":"2024-03-31T15:56:28.734169+0000","last_clean":"2024-03-31T15:56:28.734169+0000","last_became_active":"2024-03-31T15:56:26.029995+0000","last_became_peered":"2024-03-31T15:56:26.029995+0000","last_unstale":"2024-03-31T15:56:28.734169+0000","last_undegraded":"2024-03-31T15:56:28.734169+0000","last_fullsized":"2024-03-31T15:56:28.734169+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:45:21.317263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025805+0000","last_change":"2024-03-31T15:56:26.025805+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025805+0000","last_undegraded":"2024-03-31T15:56:26.025805+0000","last_fullsized":"2024-03-31T15:56:26.025805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041491+0000","last_change":"2024-03-31T15:56:26.037979+0000","last_active":"2024-03-31T15:56:26.041491+0000","last_peered":"2024-03-31T15:56:26.041491+0000","last_clean":"2024-03-31T15:56:26.041491+0000","last_became_active":"2024-03-31T15:56:26.037707+0000","last_became_peered":"2024-03-31T15:56:26.037707+0000","last_unstale":"2024-03-31T15:56:26.041491+0000","last_undegraded":"2024-03-31T15:56:26.041491+0000","last_fullsized":"2024-03-31T15:56:26.041491+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026702+0000","last_change":"2024-03-31T15:56:26.026702+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026702+0000","last_undegraded":"2024-03-31T15:56:26.026702+0000","last_fullsized":"2024-03-31T15:56:26.026702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.036608+0000","last_change":"2024-03-31T15:56:26.030797+0000","last_active":"2024-03-31T15:56:26.036608+0000","last_peered":"2024-03-31T15:56:26.036608+0000","last_clean":"2024-03-31T15:56:26.036608+0000","last_became_active":"2024-03-31T15:56:26.030470+0000","last_became_peered":"2024-03-31T15:56:26.030470+0000","last_unstale":"2024-03-31T15:56:26.036608+0000","last_undegraded":"2024-03-31T15:56:26.036608+0000","last_fullsized":"2024-03-31T15:56:26.036608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025774+0000","last_change":"2024-03-31T15:56:26.025774+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025774+0000","last_undegraded":"2024-03-31T15:56:26.025774+0000","last_fullsized":"2024-03-31T15:56:26.025774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041438+0000","last_change":"2024-03-31T15:56:26.036552+0000","last_active":"2024-03-31T15:56:26.041438+0000","last_peered":"2024-03-31T15:56:26.041438+0000","last_clean":"2024-03-31T15:56:26.041438+0000","last_became_active":"2024-03-31T15:56:26.036319+0000","last_became_peered":"2024-03-31T15:56:26.036319+0000","last_unstale":"2024-03-31T15:56:26.041438+0000","last_undegraded":"2024-03-31T15:56:26.041438+0000","last_fullsized":"2024-03-31T15:56:26.041438+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021890+0000","last_change":"2024-03-31T15:56:26.021890+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021890+0000","last_undegraded":"2024-03-31T15:56:26.021890+0000","last_fullsized":"2024-03-31T15:56:26.021890+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042115+0000","last_change":"2024-03-31T15:56:26.039110+0000","last_active":"2024-03-31T15:56:26.042115+0000","last_peered":"2024-03-31T15:56:26.042115+0000","last_clean":"2024-03-31T15:56:26.042115+0000","last_became_active":"2024-03-31T15:56:26.037645+0000","last_became_peered":"2024-03-31T15:56:26.037645+0000","last_unstale":"2024-03-31T15:56:26.042115+0000","last_undegraded":"2024-03-31T15:56:26.042115+0000","last_fullsized":"2024-03-31T15:56:26.042115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:37:11.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023537+0000","last_change":"2024-03-31T15:56:26.023537+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023537+0000","last_undegraded":"2024-03-31T15:56:26.023537+0000","last_fullsized":"2024-03-31T15:56:26.023537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048112+0000","last_change":"2024-03-31T15:56:26.040713+0000","last_active":"2024-03-31T15:56:26.048112+0000","last_peered":"2024-03-31T15:56:26.048112+0000","last_clean":"2024-03-31T15:56:26.048112+0000","last_became_active":"2024-03-31T15:56:26.040391+0000","last_became_peered":"2024-03-31T15:56:26.040391+0000","last_unstale":"2024-03-31T15:56:26.048112+0000","last_undegraded":"2024-03-31T15:56:26.048112+0000","last_fullsized":"2024-03-31T15:56:26.048112+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027172+0000","last_change":"2024-03-31T15:56:26.027172+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027172+0000","last_undegraded":"2024-03-31T15:56:26.027172+0000","last_fullsized":"2024-03-31T15:56:26.027172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041125+0000","last_change":"2024-03-31T15:56:26.035712+0000","last_active":"2024-03-31T15:56:26.041125+0000","last_peered":"2024-03-31T15:56:26.041125+0000","last_clean":"2024-03-31T15:56:26.041125+0000","last_became_active":"2024-03-31T15:56:26.034040+0000","last_became_peered":"2024-03-31T15:56:26.034040+0000","last_unstale":"2024-03-31T15:56:26.041125+0000","last_undegraded":"2024-03-31T15:56:26.041125+0000","last_fullsized":"2024-03-31T15:56:26.041125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.039713+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.037623+0000","last_became_peered":"2024-03-31T15:56:27.037623+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:10.759953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041511+0000","last_change":"2024-03-31T15:56:26.036984+0000","last_active":"2024-03-31T15:56:26.041511+0000","last_peered":"2024-03-31T15:56:26.041511+0000","last_clean":"2024-03-31T15:56:26.041511+0000","last_became_active":"2024-03-31T15:56:26.036674+0000","last_became_peered":"2024-03-31T15:56:26.036674+0000","last_unstale":"2024-03-31T15:56:26.041511+0000","last_undegraded":"2024-03-31T15:56:26.041511+0000","last_fullsized":"2024-03-31T15:56:26.041511+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048608+0000","last_change":"2024-03-31T15:56:27.037261+0000","last_active":"2024-03-31T15:56:28.048608+0000","last_peered":"2024-03-31T15:56:28.048608+0000","last_clean":"2024-03-31T15:56:28.048608+0000","last_became_active":"2024-03-31T15:56:27.036282+0000","last_became_peered":"2024-03-31T15:56:27.036282+0000","last_unstale":"2024-03-31T15:56:28.048608+0000","last_undegraded":"2024-03-31T15:56:28.048608+0000","last_fullsized":"2024-03-31T15:56:28.048608+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:25.749230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.047955+0000","last_change":"2024-03-31T15:56:26.042322+0000","last_active":"2024-03-31T15:56:26.047955+0000","last_peered":"2024-03-31T15:56:26.047955+0000","last_clean":"2024-03-31T15:56:26.047955+0000","last_became_active":"2024-03-31T15:56:26.041942+0000","last_became_peered":"2024-03-31T15:56:26.041942+0000","last_unstale":"2024-03-31T15:56:26.047955+0000","last_undegraded":"2024-03-31T15:56:26.047955+0000","last_fullsized":"2024-03-31T15:56:26.047955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022096+0000","last_change":"2024-03-31T15:56:26.022096+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022096+0000","last_undegraded":"2024-03-31T15:56:26.022096+0000","last_fullsized":"2024-03-31T15:56:26.022096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.043837+0000","last_change":"2024-03-31T15:56:26.041462+0000","last_active":"2024-03-31T15:56:26.043837+0000","last_peered":"2024-03-31T15:56:26.043837+0000","last_clean":"2024-03-31T15:56:26.043837+0000","last_became_active":"2024-03-31T15:56:26.041184+0000","last_became_peered":"2024-03-31T15:56:26.041184+0000","last_unstale":"2024-03-31T15:56:26.043837+0000","last_undegraded":"2024-03-31T15:56:26.043837+0000","last_fullsized":"2024-03-31T15:56:26.043837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.042943+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.042359+0000","last_became_peered":"2024-03-31T15:56:27.042359+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:00.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733822+0000","last_change":"2024-03-31T15:56:26.036968+0000","last_active":"2024-03-31T15:56:28.733822+0000","last_peered":"2024-03-31T15:56:28.733822+0000","last_clean":"2024-03-31T15:56:28.733822+0000","last_became_active":"2024-03-31T15:56:26.036405+0000","last_became_peered":"2024-03-31T15:56:26.036405+0000","last_unstale":"2024-03-31T15:56:28.733822+0000","last_undegraded":"2024-03-31T15:56:28.733822+0000","last_fullsized":"2024-03-31T15:56:28.733822+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:28:07.298713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733795+0000","last_change":"2024-03-31T15:56:27.057654+0000","last_active":"2024-03-31T15:56:28.733795+0000","last_peered":"2024-03-31T15:56:28.733795+0000","last_clean":"2024-03-31T15:56:28.733795+0000","last_became_active":"2024-03-31T15:56:27.057124+0000","last_became_peered":"2024-03-31T15:56:27.057124+0000","last_unstale":"2024-03-31T15:56:28.733795+0000","last_undegraded":"2024-03-31T15:56:28.733795+0000","last_fullsized":"2024-03-31T15:56:28.733795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045156+0000","last_change":"2024-03-31T15:56:26.039648+0000","last_active":"2024-03-31T15:56:26.045156+0000","last_peered":"2024-03-31T15:56:26.045156+0000","last_clean":"2024-03-31T15:56:26.045156+0000","last_became_active":"2024-03-31T15:56:26.039276+0000","last_became_peered":"2024-03-31T15:56:26.039276+0000","last_unstale":"2024-03-31T15:56:26.045156+0000","last_undegraded":"2024-03-31T15:56:26.045156+0000","last_fullsized":"2024-03-31T15:56:26.045156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047349+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:28.047349+0000","last_peered":"2024-03-31T15:56:28.047349+0000","last_clean":"2024-03-31T15:56:28.047349+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:28.047349+0000","last_undegraded":"2024-03-31T15:56:28.047349+0000","last_fullsized":"2024-03-31T15:56:28.047349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":273822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":870849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1253376,"data_stored":884868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:56:31.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-31T15:56:31.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.021551+0000', 'last_change': '2024-03-31T15:56:26.021551+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.021551+0000', 'last_undegraded': '2024-03-31T15:56:26.021551+0000', 'last_fullsized': '2024-03-31T15:56:26.021551+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:31.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.025926+0000', 'last_change': '2024-03-31T15:56:26.025926+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.025926+0000', 'last_undegraded': '2024-03-31T15:56:26.025926+0000', 'last_fullsized': '2024-03-31T15:56:26.025926+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.028615+0000', 'last_change': '2024-03-31T15:56:26.028615+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.028615+0000', 'last_undegraded': '2024-03-31T15:56:26.028615+0000', 'last_fullsized': '2024-03-31T15:56:26.028615+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.026552+0000', 'last_change': '2024-03-31T15:56:26.026552+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.026552+0000', 'last_undegraded': '2024-03-31T15:56:26.026552+0000', 'last_fullsized': '2024-03-31T15:56:26.026552+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.36 is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.022133+0000', 'last_change': '2024-03-31T15:56:26.022133+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.022133+0000', 'last_undegraded': '2024-03-31T15:56:26.022133+0000', 'last_fullsized': '2024-03-31T15:56:26.022133+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.35 is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.022163+0000', 'last_change': '2024-03-31T15:56:26.022163+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.022163+0000', 'last_undegraded': '2024-03-31T15:56:26.022163+0000', 'last_fullsized': '2024-03-31T15:56:26.022163+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-31T15:56:31.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.021075+0000', 'last_change': '2024-03-31T15:56:26.021075+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.021075+0000', 'last_undegraded': '2024-03-31T15:56:26.021075+0000', 'last_fullsized': '2024-03-31T15:56:26.021075+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.028321+0000', 'last_change': '2024-03-31T15:56:26.028321+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.028321+0000', 'last_undegraded': '2024-03-31T15:56:26.028321+0000', 'last_fullsized': '2024-03-31T15:56:26.028321+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.023949+0000', 'last_change': '2024-03-31T15:56:26.023949+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.023949+0000', 'last_undegraded': '2024-03-31T15:56:26.023949+0000', 'last_fullsized': '2024-03-31T15:56:26.023949+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.027463+0000', 'last_change': '2024-03-31T15:56:26.027463+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.027463+0000', 'last_undegraded': '2024-03-31T15:56:26.027463+0000', 'last_fullsized': '2024-03-31T15:56:26.027463+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.025451+0000', 'last_change': '2024-03-31T15:56:26.025451+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.025451+0000', 'last_undegraded': '2024-03-31T15:56:26.025451+0000', 'last_fullsized': '2024-03-31T15:56:26.025451+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.026315+0000', 'last_change': '2024-03-31T15:56:26.026315+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.026315+0000', 'last_undegraded': '2024-03-31T15:56:26.026315+0000', 'last_fullsized': '2024-03-31T15:56:26.026315+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-31T15:56:31.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.026205+0000', 'last_change': '2024-03-31T15:56:26.026205+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.026205+0000', 'last_undegraded': '2024-03-31T15:56:26.026205+0000', 'last_fullsized': '2024-03-31T15:56:26.026205+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.031242+0000', 'last_change': '2024-03-31T15:56:26.031242+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.031242+0000', 'last_undegraded': '2024-03-31T15:56:26.031242+0000', 'last_fullsized': '2024-03-31T15:56:26.031242+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.024954+0000', 'last_change': '2024-03-31T15:56:26.024954+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.024954+0000', 'last_undegraded': '2024-03-31T15:56:26.024954+0000', 'last_fullsized': '2024-03-31T15:56:26.024954+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.031116+0000', 'last_change': '2024-03-31T15:56:26.031116+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.031116+0000', 'last_undegraded': '2024-03-31T15:56:26.031116+0000', 'last_fullsized': '2024-03-31T15:56:26.031116+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.027162+0000', 'last_change': '2024-03-31T15:56:26.027162+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.027162+0000', 'last_undegraded': '2024-03-31T15:56:26.027162+0000', 'last_fullsized': '2024-03-31T15:56:26.027162+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-31T15:56:31.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.026382+0000', 'last_change': '2024-03-31T15:56:26.026382+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.026382+0000', 'last_undegraded': '2024-03-31T15:56:26.026382+0000', 'last_fullsized': '2024-03-31T15:56:26.026382+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.027580+0000', 'last_change': '2024-03-31T15:56:26.027580+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.027580+0000', 'last_undegraded': '2024-03-31T15:56:26.027580+0000', 'last_fullsized': '2024-03-31T15:56:26.027580+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.025656+0000', 'last_change': '2024-03-31T15:56:26.025656+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.025656+0000', 'last_undegraded': '2024-03-31T15:56:26.025656+0000', 'last_fullsized': '2024-03-31T15:56:26.025656+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.023782+0000', 'last_change': '2024-03-31T15:56:26.023782+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.023782+0000', 'last_undegraded': '2024-03-31T15:56:26.023782+0000', 'last_fullsized': '2024-03-31T15:56:26.023782+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-03-31T15:56:31.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.023933+0000', 'last_change': '2024-03-31T15:56:26.023933+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.023933+0000', 'last_undegraded': '2024-03-31T15:56:26.023933+0000', 'last_fullsized': '2024-03-31T15:56:26.023933+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.022785+0000', 'last_change': '2024-03-31T15:56:26.022785+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.022785+0000', 'last_undegraded': '2024-03-31T15:56:26.022785+0000', 'last_fullsized': '2024-03-31T15:56:26.022785+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.024021+0000', 'last_change': '2024-03-31T15:56:26.024021+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.024021+0000', 'last_undegraded': '2024-03-31T15:56:26.024021+0000', 'last_fullsized': '2024-03-31T15:56:26.024021+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.027856+0000', 'last_change': '2024-03-31T15:56:26.027856+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.027856+0000', 'last_undegraded': '2024-03-31T15:56:26.027856+0000', 'last_fullsized': '2024-03-31T15:56:26.027856+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.022357+0000', 'last_change': '2024-03-31T15:56:26.022357+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.022357+0000', 'last_undegraded': '2024-03-31T15:56:26.022357+0000', 'last_fullsized': '2024-03-31T15:56:26.022357+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.025805+0000', 'last_change': '2024-03-31T15:56:26.025805+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.025805+0000', 'last_undegraded': '2024-03-31T15:56:26.025805+0000', 'last_fullsized': '2024-03-31T15:56:26.025805+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:32.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.026702+0000', 'last_change': '2024-03-31T15:56:26.026702+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.026702+0000', 'last_undegraded': '2024-03-31T15:56:26.026702+0000', 'last_fullsized': '2024-03-31T15:56:26.026702+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.025774+0000', 'last_change': '2024-03-31T15:56:26.025774+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.025774+0000', 'last_undegraded': '2024-03-31T15:56:26.025774+0000', 'last_fullsized': '2024-03-31T15:56:26.025774+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.021890+0000', 'last_change': '2024-03-31T15:56:26.021890+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.021890+0000', 'last_undegraded': '2024-03-31T15:56:26.021890+0000', 'last_fullsized': '2024-03-31T15:56:26.021890+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.023537+0000', 'last_change': '2024-03-31T15:56:26.023537+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.023537+0000', 'last_undegraded': '2024-03-31T15:56:26.023537+0000', 'last_fullsized': '2024-03-31T15:56:26.023537+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.027172+0000', 'last_change': '2024-03-31T15:56:26.027172+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.027172+0000', 'last_undegraded': '2024-03-31T15:56:26.027172+0000', 'last_fullsized': '2024-03-31T15:56:26.027172+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-03-31T15:56:32.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-03-31T15:56:26.022096+0000', 'last_change': '2024-03-31T15:56:26.022096+0000', 'last_active': '2024-03-31T15:56:26.009477+0000', 'last_peered': '2024-03-31T15:56:26.009477+0000', 'last_clean': '2024-03-31T15:56:26.009477+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-31T15:56:26.022096+0000', 'last_undegraded': '2024-03-31T15:56:26.022096+0000', 'last_fullsized': '2024-03-31T15:56:26.022096+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'last_clean_scrub_stamp': '2024-03-31T15:56:26.009477+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-31T15:56:32.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-31T15:56:32.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.172+0000 7f6eef600640 1 Processor -- start 2024-03-31T15:56:32.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.172+0000 7f6eef600640 1 -- start start 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eef600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eef600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ee806d7c0 0x7f6ee815e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eef600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee815f780 con 0x7f6ee806c7b0 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eef600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f6ee81622b0 con 0x7f6ee806d7c0 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eef600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee815f1c0 con 0x7f6ee815e1b0 2024-03-31T15:56:32.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ee806d7c0 0x7f6ee815e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f6ee806d7c0 0x7f6ee815e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41958/0 (socket says 172.21.15.90:41958) 2024-03-31T15:56:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6ee7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6ee806c7b0 0x7f6ee806d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50466/0 (socket says 172.21.15.90:50466) 2024-03-31T15:56:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 -- 172.21.15.90:0/1400919624 learned_addr learned my addr 172.21.15.90:0/1400919624 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eecc00640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 -- 172.21.15.90:0/1400919624 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 msgr2=0x7f6ee8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:32.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee8162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 -- 172.21.15.90:0/1400919624 >> v1:172.21.15.90:6789/0 conn(0x7f6ee806c7b0 legacy=0x7f6ee806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.176+0000 7f6eed600640 1 -- 172.21.15.90:0/1400919624 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee815f4a0 con 0x7f6ee806d7c0 2024-03-31T15:56:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eed600640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806d7c0 0x7f6ee815e590 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009e50 tx=0x7f6ee001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdf1585b68028fa server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6ee7400640 1 -- 172.21.15.90:0/1400919624 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00a5830 con 0x7f6ee806d7c0 2024-03-31T15:56:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6ee7400640 1 -- 172.21.15.90:0/1400919624 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee00a59d0 con 0x7f6ee806d7c0 2024-03-31T15:56:32.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6ee7400640 1 -- 172.21.15.90:0/1400919624 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00a5da0 con 0x7f6ee806d7c0 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- 172.21.15.90:0/1400919624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806d7c0 msgr2=0x7f6ee815e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806d7c0 0x7f6ee815e590 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009e50 tx=0x7f6ee001c970 comp rx=0 tx=0).stop 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- 172.21.15.90:0/1400919624 shutdown_connections 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806d7c0 0x7f6ee815e590 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- 172.21.15.90:0/1400919624 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- 172.21.15.90:0/1400919624 >> 172.21.15.90:0/1400919624 conn(0x7f6ee809aa40 msgr2=0x7f6ee809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- 172.21.15.90:0/1400919624 shutdown_connections 2024-03-31T15:56:32.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- 172.21.15.90:0/1400919624 wait complete. 2024-03-31T15:56:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 Processor -- start 2024-03-31T15:56:32.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- start start 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ee806d7c0 0x7f6ee8155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee80afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee80d2940 con 0x7f6ee806c7b0 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee80d27c0 con 0x7f6ee806d7c0 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eef600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee80d2ac0 con 0x7f6ee815e1b0 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eecc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eecc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41970/0 (socket says 172.21.15.90:41970) 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.180+0000 7f6eecc00640 1 -- 172.21.15.90:0/608576020 learned_addr learned my addr 172.21.15.90:0/608576020 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eed600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee80afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6ee7e00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ee806d7c0 0x7f6ee8155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 -- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 msgr2=0x7f6ee80afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee80afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 -- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ee806d7c0 msgr2=0x7f6ee8155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ee806d7c0 0x7f6ee8155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 -- 172.21.15.90:0/608576020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee80b1720 con 0x7f6ee806c7b0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eecc00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00d800 tx=0x7f6edc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00b6c0 con 0x7f6ee806c7b0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6edc019070 con 0x7f6ee806c7b0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee80c7b90 con 0x7f6ee806c7b0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00ba60 con 0x7f6ee806c7b0 2024-03-31T15:56:32.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ee80c8150 con 0x7f6ee806c7b0 2024-03-31T15:56:32.179 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.184+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ee80686d0 con 0x7f6ee806c7b0 2024-03-31T15:56:32.180 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.188+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6edc00bc00 con 0x7f6ee806c7b0 2024-03-31T15:56:32.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.188+0000 7f6ee4c00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 0x7f6ec806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.188+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f6edc086f90 con 0x7f6ee806c7b0 2024-03-31T15:56:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.188+0000 7f6ee7e00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 0x7f6ec806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.188+0000 7f6ee7e00640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 0x7f6ec806a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6ee81602c0 tx=0x7f6ed4007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.192+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6edc0573b0 con 0x7f6ee806c7b0 2024-03-31T15:56:32.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6ee80b1b60 con 0x7f6ee806c7b0 2024-03-31T15:56:32.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6ee4c00640 1 -- 172.21.15.90:0/608576020 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1260 (secure 0 0 0) 0x7f6edc056cb0 con 0x7f6ee806c7b0 2024-03-31T15:56:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:32.401 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711900415,"num_in_osds":8,"osd_in_since":1711900406,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":64},{"state_name":"creating+peering","count":33}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":590514,"bytes_used":227647488,"bytes_avail":772866465792,"bytes_total":773094113280,"inactive_pgs_ratio":0.34020617604255676,"write_bytes_sec":1535,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":27,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":6213}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-03-31T15:56:13.250944+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 msgr2=0x7f6ec806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 0x7f6ec806a840 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6ee81602c0 tx=0x7f6ed4007cf0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 msgr2=0x7f6ee80c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00d800 tx=0x7f6edc00dcc0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 shutdown_connections 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6ec8068380 0x7f6ec806a840 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ee815e1b0 0x7f6ee80afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ee806d7c0 0x7f6ee8155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 --2- 172.21.15.90:0/608576020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ee806c7b0 0x7f6ee80c8950 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 >> 172.21.15.90:0/608576020 conn(0x7f6ee809aa40 msgr2=0x7f6ee80d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 shutdown_connections 2024-03-31T15:56:32.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.408+0000 7f6eef600640 1 -- 172.21.15.90:0/608576020 wait complete. 2024-03-31T15:56:32.423 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-31T15:56:32.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:56:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.564+0000 7f1b06600640 1 Processor -- start 2024-03-31T15:56:32.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.564+0000 7f1b06600640 1 -- start start 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1b06600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b000d0490 0x7f1b0006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1b06600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b0015e8a0 con 0x7f1b000d48d0 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1b06600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b0006cf90 con 0x7f1b0015e4c0 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1b06600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1b0006d270 con 0x7f1b000d0490 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1af7e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1b000d48d0 0x7f1b0006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50468/0 (socket says 172.21.15.90:50468) 2024-03-31T15:56:32.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1af7e00640 1 -- 172.21.15.90:0/3940511014 learned_addr learned my addr 172.21.15.90:0/3940511014 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:32.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> v2:172.21.15.102:3300/0 conn(0x7f1b000d0490 0x7f1b0006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945030912 0 0) 0x7f1b0015e8a0 con 0x7f1b000d48d0 2024-03-31T15:56:32.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ae4003530 con 0x7f1b000d48d0 2024-03-31T15:56:32.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949727169 0 0) 0x7f1b0006cf90 con 0x7f1b0015e4c0 2024-03-31T15:56:32.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b0015e8a0 con 0x7f1b0015e4c0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 989833885 0 0) 0x7f1ae4003530 con 0x7f1b000d48d0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 >> v2:172.21.15.102:3300/0 conn(0x7f1b000d0490 msgr2=0x7f1b0006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 --2- 172.21.15.90:0/3940511014 >> v2:172.21.15.102:3300/0 conn(0x7f1b000d0490 0x7f1b0006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 >> v1:172.21.15.102:6789/0 conn(0x7f1b0015e4c0 legacy=0x7f1b000d41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0006d550 con 0x7f1b000d48d0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f1ae80016a0 con 0x7f1b000d48d0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> v2:172.21.15.102:3300/0 conn(0x7f1b000d0490 0x7f1b0006c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 >> v1:172.21.15.90:6789/0 conn(0x7f1b000d48d0 legacy=0x7f1b0006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4003ba0 0x7f1ae4003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae4004810 0x7f1ae4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 0x7f1ae400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0006d270 con 0x7f1ae40051d0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae400c0a0 con 0x7f1ae4003ba0 2024-03-31T15:56:32.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae400c2a0 con 0x7f1ae4004810 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 0x7f1ae400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1aff400640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4003ba0 0x7f1ae4003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1af7e00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae4004810 0x7f1ae4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 -- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae4004810 msgr2=0x7f1ae4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae4004810 0x7f1ae4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 -- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4003ba0 msgr2=0x7f1ae4003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4003ba0 0x7f1ae4003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.568+0000 7f1affe00640 1 -- 172.21.15.90:0/3940511014 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae4003530 con 0x7f1ae40051d0 2024-03-31T15:56:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1affe00640 1 -- 172.21.15.90:0/3940511014 reap_dead start 2024-03-31T15:56:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1affe00640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 0x7f1ae400b9d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1af800e350 tx=0x7f1af801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6057b545bb1ea1a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8009de0 con 0x7f1ae40051d0 2024-03-31T15:56:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af80a5cf0 con 0x7f1ae40051d0 2024-03-31T15:56:32.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1afea00640 1 -- 172.21.15.90:0/3940511014 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8003be0 con 0x7f1ae40051d0 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 msgr2=0x7f1ae400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 0x7f1ae400b9d0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f1af800e350 tx=0x7f1af801cca0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- 172.21.15.90:0/3940511014 shutdown_connections 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 --2- 172.21.15.90:0/3940511014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1ae40051d0 0x7f1ae400b9d0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- 172.21.15.90:0/3940511014 >> 172.21.15.90:0/3940511014 conn(0x7f1b0008db60 msgr2=0x7f1b0009d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- 172.21.15.90:0/3940511014 shutdown_connections 2024-03-31T15:56:32.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- 172.21.15.90:0/3940511014 wait complete. 2024-03-31T15:56:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 Processor -- start 2024-03-31T15:56:32.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.572+0000 7f1b06600640 1 -- start start 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae40051d0 0x7f1b0015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b000b7ec0 0x7f1b000b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1b000d41c0 con 0x7f1b000b8320 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1b000ceef0 con 0x7f1b000b7ec0 2024-03-31T15:56:32.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1b000d4340 con 0x7f1ae40051d0 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1aff400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae40051d0 0x7f1b0015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41988/0 (socket says 172.21.15.90:41988) 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 -- 172.21.15.90:0/2699090461 learned_addr learned my addr 172.21.15.90:0/2699090461 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1af7e00640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b000b7ec0 0x7f1b000b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 -- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae40051d0 msgr2=0x7f1b0015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae40051d0 0x7f1b0015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 -- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b000b7ec0 msgr2=0x7f1b000b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b000b7ec0 0x7f1b000b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 -- 172.21.15.90:0/2699090461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b000b6e20 con 0x7f1b000b8320 2024-03-31T15:56:32.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1affe00640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1af801cad0 tx=0x7f1af8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8009de0 con 0x7f1b000b8320 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af80a5ab0 con 0x7f1b000b8320 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b000b7100 con 0x7f1b000b8320 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af80039a0 con 0x7f1b000b8320 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b000b76c0 con 0x7f1b000b8320 2024-03-31T15:56:32.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.576+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b000686d0 con 0x7f1b000b8320 2024-03-31T15:56:32.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.580+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1af80c0030 con 0x7f1b000b8320 2024-03-31T15:56:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.580+0000 7f1af7400640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 0x7f1adc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.580+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f1af8121ab0 con 0x7f1b000b8320 2024-03-31T15:56:32.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.580+0000 7f1aff400640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 0x7f1adc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:32.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.580+0000 7f1aff400640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 0x7f1adc06a840 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1af0000c10 tx=0x7f1af0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:32.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.584+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1ae8007e20 con 0x7f1b000b8320 2024-03-31T15:56:32.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.736+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b000d1680 con 0x7f1adc068380 2024-03-31T15:56:32.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.744+0000 7f1af7400640 1 -- 172.21.15.90:0/2699090461 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249206 (secure 0 0 0) 0x7f1b000d1680 con 0x7f1adc068380 2024-03-31T15:56:32.740 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:32.741 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:56:32.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 msgr2=0x7f1adc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 0x7f1adc06a840 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1af0000c10 tx=0x7f1af0009040 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 msgr2=0x7f1b000b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1af801cad0 tx=0x7f1af8009750 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 shutdown_connections 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1adc068380 0x7f1adc06a840 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1b000b8320 0x7f1b000b8eb0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b000b7ec0 0x7f1b000b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 --2- 172.21.15.90:0/2699090461 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1ae40051d0 0x7f1b0015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 >> 172.21.15.90:0/2699090461 conn(0x7f1b0008db60 msgr2=0x7f1b0006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 shutdown_connections 2024-03-31T15:56:32.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:32.748+0000 7f1b06600640 1 -- 172.21.15.90:0/2699090461 wait complete. 2024-03-31T15:56:32.762 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-31T15:56:31.255055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590514,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":180,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222312,"kb_used_data":6140,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752408,"statfs":{"total":773094113280,"available":772866465792,"internally_reserved":0,"allocated":6287360,"data_stored":3459774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001246"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732549+0000","last_change":"2024-03-31T15:56:27.060874+0000","last_active":"2024-03-31T15:56:28.732549+0000","last_peered":"2024-03-31T15:56:28.732549+0000","last_clean":"2024-03-31T15:56:28.732549+0000","last_became_active":"2024-03-31T15:56:27.060630+0000","last_became_peered":"2024-03-31T15:56:27.060630+0000","last_unstale":"2024-03-31T15:56:28.732549+0000","last_undegraded":"2024-03-31T15:56:28.732549+0000","last_fullsized":"2024-03-31T15:56:28.732549+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:56:54.783818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734232+0000","last_change":"2024-03-31T15:56:27.040029+0000","last_active":"2024-03-31T15:56:28.734232+0000","last_peered":"2024-03-31T15:56:28.734232+0000","last_clean":"2024-03-31T15:56:28.734232+0000","last_became_active":"2024-03-31T15:56:27.036544+0000","last_became_peered":"2024-03-31T15:56:27.036544+0000","last_unstale":"2024-03-31T15:56:28.734232+0000","last_undegraded":"2024-03-31T15:56:28.734232+0000","last_fullsized":"2024-03-31T15:56:28.734232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:57.935165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048573+0000","last_change":"2024-03-31T15:56:27.040254+0000","last_active":"2024-03-31T15:56:28.048573+0000","last_peered":"2024-03-31T15:56:28.048573+0000","last_clean":"2024-03-31T15:56:28.048573+0000","last_became_active":"2024-03-31T15:56:27.039839+0000","last_became_peered":"2024-03-31T15:56:27.039839+0000","last_unstale":"2024-03-31T15:56:28.048573+0000","last_undegraded":"2024-03-31T15:56:28.048573+0000","last_fullsized":"2024-03-31T15:56:28.048573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:38.034420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021551+0000","last_change":"2024-03-31T15:56:26.021551+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021551+0000","last_undegraded":"2024-03-31T15:56:26.021551+0000","last_fullsized":"2024-03-31T15:56:26.021551+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731537+0000","last_change":"2024-03-31T15:56:27.057953+0000","last_active":"2024-03-31T15:56:28.731537+0000","last_peered":"2024-03-31T15:56:28.731537+0000","last_clean":"2024-03-31T15:56:28.731537+0000","last_became_active":"2024-03-31T15:56:27.057680+0000","last_became_peered":"2024-03-31T15:56:27.057680+0000","last_unstale":"2024-03-31T15:56:28.731537+0000","last_undegraded":"2024-03-31T15:56:28.731537+0000","last_fullsized":"2024-03-31T15:56:28.731537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:41:20.883103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025926+0000","last_change":"2024-03-31T15:56:26.025926+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025926+0000","last_undegraded":"2024-03-31T15:56:26.025926+0000","last_fullsized":"2024-03-31T15:56:26.025926+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028615+0000","last_change":"2024-03-31T15:56:26.028615+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028615+0000","last_undegraded":"2024-03-31T15:56:26.028615+0000","last_fullsized":"2024-03-31T15:56:26.028615+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733707+0000","last_change":"2024-03-31T15:56:27.040713+0000","last_active":"2024-03-31T15:56:28.733707+0000","last_peered":"2024-03-31T15:56:28.733707+0000","last_clean":"2024-03-31T15:56:28.733707+0000","last_became_active":"2024-03-31T15:56:27.040439+0000","last_became_peered":"2024-03-31T15:56:27.040439+0000","last_unstale":"2024-03-31T15:56:28.733707+0000","last_undegraded":"2024-03-31T15:56:28.733707+0000","last_fullsized":"2024-03-31T15:56:28.733707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026552+0000","last_change":"2024-03-31T15:56:26.026552+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026552+0000","last_undegraded":"2024-03-31T15:56:26.026552+0000","last_fullsized":"2024-03-31T15:56:26.026552+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022133+0000","last_change":"2024-03-31T15:56:26.022133+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022133+0000","last_undegraded":"2024-03-31T15:56:26.022133+0000","last_fullsized":"2024-03-31T15:56:26.022133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022163+0000","last_change":"2024-03-31T15:56:26.022163+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022163+0000","last_undegraded":"2024-03-31T15:56:26.022163+0000","last_fullsized":"2024-03-31T15:56:26.022163+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021075+0000","last_change":"2024-03-31T15:56:26.021075+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021075+0000","last_undegraded":"2024-03-31T15:56:26.021075+0000","last_fullsized":"2024-03-31T15:56:26.021075+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.028321+0000","last_change":"2024-03-31T15:56:26.028321+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.028321+0000","last_undegraded":"2024-03-31T15:56:26.028321+0000","last_fullsized":"2024-03-31T15:56:26.028321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023949+0000","last_change":"2024-03-31T15:56:26.023949+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023949+0000","last_undegraded":"2024-03-31T15:56:26.023949+0000","last_fullsized":"2024-03-31T15:56:26.023949+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027463+0000","last_change":"2024-03-31T15:56:26.027463+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027463+0000","last_undegraded":"2024-03-31T15:56:26.027463+0000","last_fullsized":"2024-03-31T15:56:26.027463+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025451+0000","last_change":"2024-03-31T15:56:26.025451+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025451+0000","last_undegraded":"2024-03-31T15:56:26.025451+0000","last_fullsized":"2024-03-31T15:56:26.025451+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734398+0000","last_change":"2024-03-31T15:56:27.041513+0000","last_active":"2024-03-31T15:56:28.734398+0000","last_peered":"2024-03-31T15:56:28.734398+0000","last_clean":"2024-03-31T15:56:28.734398+0000","last_became_active":"2024-03-31T15:56:27.041210+0000","last_became_peered":"2024-03-31T15:56:27.041210+0000","last_unstale":"2024-03-31T15:56:28.734398+0000","last_undegraded":"2024-03-31T15:56:28.734398+0000","last_fullsized":"2024-03-31T15:56:28.734398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:55.146299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048591+0000","last_change":"2024-03-31T15:56:27.049282+0000","last_active":"2024-03-31T15:56:28.048591+0000","last_peered":"2024-03-31T15:56:28.048591+0000","last_clean":"2024-03-31T15:56:28.048591+0000","last_became_active":"2024-03-31T15:56:27.048536+0000","last_became_peered":"2024-03-31T15:56:27.048536+0000","last_unstale":"2024-03-31T15:56:28.048591+0000","last_undegraded":"2024-03-31T15:56:28.048591+0000","last_fullsized":"2024-03-31T15:56:28.048591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:49:37.697269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732485+0000","last_change":"2024-03-31T15:56:27.060292+0000","last_active":"2024-03-31T15:56:28.732485+0000","last_peered":"2024-03-31T15:56:28.732485+0000","last_clean":"2024-03-31T15:56:28.732485+0000","last_became_active":"2024-03-31T15:56:27.060044+0000","last_became_peered":"2024-03-31T15:56:27.060044+0000","last_unstale":"2024-03-31T15:56:28.732485+0000","last_undegraded":"2024-03-31T15:56:28.732485+0000","last_fullsized":"2024-03-31T15:56:28.732485+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:03.617564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734311+0000","last_change":"2024-03-31T15:56:27.059752+0000","last_active":"2024-03-31T15:56:28.734311+0000","last_peered":"2024-03-31T15:56:28.734311+0000","last_clean":"2024-03-31T15:56:28.734311+0000","last_became_active":"2024-03-31T15:56:27.059504+0000","last_became_peered":"2024-03-31T15:56:27.059504+0000","last_unstale":"2024-03-31T15:56:28.734311+0000","last_undegraded":"2024-03-31T15:56:28.734311+0000","last_fullsized":"2024-03-31T15:56:28.734311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:03.532453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048464+0000","last_change":"2024-03-31T15:56:27.030882+0000","last_active":"2024-03-31T15:56:28.048464+0000","last_peered":"2024-03-31T15:56:28.048464+0000","last_clean":"2024-03-31T15:56:28.048464+0000","last_became_active":"2024-03-31T15:56:27.030222+0000","last_became_peered":"2024-03-31T15:56:27.030222+0000","last_unstale":"2024-03-31T15:56:28.048464+0000","last_undegraded":"2024-03-31T15:56:28.048464+0000","last_fullsized":"2024-03-31T15:56:28.048464+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:25.459268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733876+0000","last_change":"2024-03-31T15:56:27.053182+0000","last_active":"2024-03-31T15:56:28.733876+0000","last_peered":"2024-03-31T15:56:28.733876+0000","last_clean":"2024-03-31T15:56:28.733876+0000","last_became_active":"2024-03-31T15:56:27.052882+0000","last_became_peered":"2024-03-31T15:56:27.052882+0000","last_unstale":"2024-03-31T15:56:28.733876+0000","last_undegraded":"2024-03-31T15:56:28.733876+0000","last_fullsized":"2024-03-31T15:56:28.733876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047955+0000","last_change":"2024-03-31T15:56:27.037326+0000","last_active":"2024-03-31T15:56:28.047955+0000","last_peered":"2024-03-31T15:56:28.047955+0000","last_clean":"2024-03-31T15:56:28.047955+0000","last_became_active":"2024-03-31T15:56:27.036842+0000","last_became_peered":"2024-03-31T15:56:27.036842+0000","last_unstale":"2024-03-31T15:56:28.047955+0000","last_undegraded":"2024-03-31T15:56:28.047955+0000","last_fullsized":"2024-03-31T15:56:28.047955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:13.092644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026315+0000","last_change":"2024-03-31T15:56:26.026315+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026315+0000","last_undegraded":"2024-03-31T15:56:26.026315+0000","last_fullsized":"2024-03-31T15:56:26.026315+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732648+0000","last_change":"2024-03-31T15:56:27.058506+0000","last_active":"2024-03-31T15:56:28.732648+0000","last_peered":"2024-03-31T15:56:28.732648+0000","last_clean":"2024-03-31T15:56:28.732648+0000","last_became_active":"2024-03-31T15:56:27.058252+0000","last_became_peered":"2024-03-31T15:56:27.058252+0000","last_unstale":"2024-03-31T15:56:28.732648+0000","last_undegraded":"2024-03-31T15:56:28.732648+0000","last_fullsized":"2024-03-31T15:56:28.732648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:03.064057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026205+0000","last_change":"2024-03-31T15:56:26.026205+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026205+0000","last_undegraded":"2024-03-31T15:56:26.026205+0000","last_fullsized":"2024-03-31T15:56:26.026205+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031242+0000","last_change":"2024-03-31T15:56:26.031242+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031242+0000","last_undegraded":"2024-03-31T15:56:26.031242+0000","last_fullsized":"2024-03-31T15:56:26.031242+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732586+0000","last_change":"2024-03-31T15:56:27.058392+0000","last_active":"2024-03-31T15:56:28.732586+0000","last_peered":"2024-03-31T15:56:28.732586+0000","last_clean":"2024-03-31T15:56:28.732586+0000","last_became_active":"2024-03-31T15:56:27.058103+0000","last_became_peered":"2024-03-31T15:56:27.058103+0000","last_unstale":"2024-03-31T15:56:28.732586+0000","last_undegraded":"2024-03-31T15:56:28.732586+0000","last_fullsized":"2024-03-31T15:56:28.732586+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:38.875392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734201+0000","last_change":"2024-03-31T15:56:27.041576+0000","last_active":"2024-03-31T15:56:28.734201+0000","last_peered":"2024-03-31T15:56:28.734201+0000","last_clean":"2024-03-31T15:56:28.734201+0000","last_became_active":"2024-03-31T15:56:27.041327+0000","last_became_peered":"2024-03-31T15:56:27.041327+0000","last_unstale":"2024-03-31T15:56:28.734201+0000","last_undegraded":"2024-03-31T15:56:28.734201+0000","last_fullsized":"2024-03-31T15:56:28.734201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:39.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048359+0000","last_change":"2024-03-31T15:56:27.034473+0000","last_active":"2024-03-31T15:56:28.048359+0000","last_peered":"2024-03-31T15:56:28.048359+0000","last_clean":"2024-03-31T15:56:28.048359+0000","last_became_active":"2024-03-31T15:56:27.034209+0000","last_became_peered":"2024-03-31T15:56:27.034209+0000","last_unstale":"2024-03-31T15:56:28.048359+0000","last_undegraded":"2024-03-31T15:56:28.048359+0000","last_fullsized":"2024-03-31T15:56:28.048359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:45.169835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731505+0000","last_change":"2024-03-31T15:56:27.058056+0000","last_active":"2024-03-31T15:56:28.731505+0000","last_peered":"2024-03-31T15:56:28.731505+0000","last_clean":"2024-03-31T15:56:28.731505+0000","last_became_active":"2024-03-31T15:56:27.057786+0000","last_became_peered":"2024-03-31T15:56:27.057786+0000","last_unstale":"2024-03-31T15:56:28.731505+0000","last_undegraded":"2024-03-31T15:56:28.731505+0000","last_fullsized":"2024-03-31T15:56:28.731505+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:04:01.295156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024954+0000","last_change":"2024-03-31T15:56:26.024954+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024954+0000","last_undegraded":"2024-03-31T15:56:26.024954+0000","last_fullsized":"2024-03-31T15:56:26.024954+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.031116+0000","last_change":"2024-03-31T15:56:26.031116+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.031116+0000","last_undegraded":"2024-03-31T15:56:26.031116+0000","last_fullsized":"2024-03-31T15:56:26.031116+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044934+0000","last_change":"2024-03-31T15:56:26.043587+0000","last_active":"2024-03-31T15:56:26.044934+0000","last_peered":"2024-03-31T15:56:26.044934+0000","last_clean":"2024-03-31T15:56:26.044934+0000","last_became_active":"2024-03-31T15:56:26.043321+0000","last_became_peered":"2024-03-31T15:56:26.043321+0000","last_unstale":"2024-03-31T15:56:26.044934+0000","last_undegraded":"2024-03-31T15:56:26.044934+0000","last_fullsized":"2024-03-31T15:56:26.044934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027162+0000","last_change":"2024-03-31T15:56:26.027162+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027162+0000","last_undegraded":"2024-03-31T15:56:26.027162+0000","last_fullsized":"2024-03-31T15:56:26.027162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734183+0000","last_change":"2024-03-31T15:56:26.029688+0000","last_active":"2024-03-31T15:56:28.734183+0000","last_peered":"2024-03-31T15:56:28.734183+0000","last_clean":"2024-03-31T15:56:28.734183+0000","last_became_active":"2024-03-31T15:56:26.029396+0000","last_became_peered":"2024-03-31T15:56:26.029396+0000","last_unstale":"2024-03-31T15:56:28.734183+0000","last_undegraded":"2024-03-31T15:56:28.734183+0000","last_fullsized":"2024-03-31T15:56:28.734183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:52:46.291141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026382+0000","last_change":"2024-03-31T15:56:26.026382+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026382+0000","last_undegraded":"2024-03-31T15:56:26.026382+0000","last_fullsized":"2024-03-31T15:56:26.026382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045804+0000","last_change":"2024-03-31T15:56:26.044595+0000","last_active":"2024-03-31T15:56:26.045804+0000","last_peered":"2024-03-31T15:56:26.045804+0000","last_clean":"2024-03-31T15:56:26.045804+0000","last_became_active":"2024-03-31T15:56:26.044321+0000","last_became_peered":"2024-03-31T15:56:26.044321+0000","last_unstale":"2024-03-31T15:56:26.045804+0000","last_undegraded":"2024-03-31T15:56:26.045804+0000","last_fullsized":"2024-03-31T15:56:26.045804+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027580+0000","last_change":"2024-03-31T15:56:26.027580+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027580+0000","last_undegraded":"2024-03-31T15:56:26.027580+0000","last_fullsized":"2024-03-31T15:56:26.027580+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.038619+0000","last_change":"2024-03-31T15:56:26.035102+0000","last_active":"2024-03-31T15:56:26.038619+0000","last_peered":"2024-03-31T15:56:26.038619+0000","last_clean":"2024-03-31T15:56:26.038619+0000","last_became_active":"2024-03-31T15:56:26.034821+0000","last_became_peered":"2024-03-31T15:56:26.034821+0000","last_unstale":"2024-03-31T15:56:26.038619+0000","last_undegraded":"2024-03-31T15:56:26.038619+0000","last_fullsized":"2024-03-31T15:56:26.038619+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:48.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733891+0000","last_change":"2024-03-31T15:56:27.053152+0000","last_active":"2024-03-31T15:56:28.733891+0000","last_peered":"2024-03-31T15:56:28.733891+0000","last_clean":"2024-03-31T15:56:28.733891+0000","last_became_active":"2024-03-31T15:56:27.052870+0000","last_became_peered":"2024-03-31T15:56:27.052870+0000","last_unstale":"2024-03-31T15:56:28.733891+0000","last_undegraded":"2024-03-31T15:56:28.733891+0000","last_fullsized":"2024-03-31T15:56:28.733891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:53.273112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042800+0000","last_change":"2024-03-31T15:56:26.035499+0000","last_active":"2024-03-31T15:56:26.042800+0000","last_peered":"2024-03-31T15:56:26.042800+0000","last_clean":"2024-03-31T15:56:26.042800+0000","last_became_active":"2024-03-31T15:56:26.035216+0000","last_became_peered":"2024-03-31T15:56:26.035216+0000","last_unstale":"2024-03-31T15:56:26.042800+0000","last_undegraded":"2024-03-31T15:56:26.042800+0000","last_fullsized":"2024-03-31T15:56:26.042800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734108+0000","last_change":"2024-03-31T15:56:27.055018+0000","last_active":"2024-03-31T15:56:28.734108+0000","last_peered":"2024-03-31T15:56:28.734108+0000","last_clean":"2024-03-31T15:56:28.734108+0000","last_became_active":"2024-03-31T15:56:27.054765+0000","last_became_peered":"2024-03-31T15:56:27.054765+0000","last_unstale":"2024-03-31T15:56:28.734108+0000","last_undegraded":"2024-03-31T15:56:28.734108+0000","last_fullsized":"2024-03-31T15:56:28.734108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:49.516522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.044103+0000","last_change":"2024-03-31T15:56:26.040614+0000","last_active":"2024-03-31T15:56:26.044103+0000","last_peered":"2024-03-31T15:56:26.044103+0000","last_clean":"2024-03-31T15:56:26.044103+0000","last_became_active":"2024-03-31T15:56:26.040371+0000","last_became_peered":"2024-03-31T15:56:26.040371+0000","last_unstale":"2024-03-31T15:56:26.044103+0000","last_undegraded":"2024-03-31T15:56:26.044103+0000","last_fullsized":"2024-03-31T15:56:26.044103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025656+0000","last_change":"2024-03-31T15:56:26.025656+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025656+0000","last_undegraded":"2024-03-31T15:56:26.025656+0000","last_fullsized":"2024-03-31T15:56:26.025656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041171+0000","last_change":"2024-03-31T15:56:26.037304+0000","last_active":"2024-03-31T15:56:26.041171+0000","last_peered":"2024-03-31T15:56:26.041171+0000","last_clean":"2024-03-31T15:56:26.041171+0000","last_became_active":"2024-03-31T15:56:26.037038+0000","last_became_peered":"2024-03-31T15:56:26.037038+0000","last_unstale":"2024-03-31T15:56:26.041171+0000","last_undegraded":"2024-03-31T15:56:26.041171+0000","last_fullsized":"2024-03-31T15:56:26.041171+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:52.264596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733928+0000","last_change":"2024-03-31T15:56:27.043180+0000","last_active":"2024-03-31T15:56:28.733928+0000","last_peered":"2024-03-31T15:56:28.733928+0000","last_clean":"2024-03-31T15:56:28.733928+0000","last_became_active":"2024-03-31T15:56:27.042692+0000","last_became_peered":"2024-03-31T15:56:27.042692+0000","last_unstale":"2024-03-31T15:56:28.733928+0000","last_undegraded":"2024-03-31T15:56:28.733928+0000","last_fullsized":"2024-03-31T15:56:28.733928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:21:35.971485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045526+0000","last_change":"2024-03-31T15:56:26.040038+0000","last_active":"2024-03-31T15:56:26.045526+0000","last_peered":"2024-03-31T15:56:26.045526+0000","last_clean":"2024-03-31T15:56:26.045526+0000","last_became_active":"2024-03-31T15:56:26.039654+0000","last_became_peered":"2024-03-31T15:56:26.039654+0000","last_unstale":"2024-03-31T15:56:26.045526+0000","last_undegraded":"2024-03-31T15:56:26.045526+0000","last_fullsized":"2024-03-31T15:56:26.045526+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023782+0000","last_change":"2024-03-31T15:56:26.023782+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023782+0000","last_undegraded":"2024-03-31T15:56:26.023782+0000","last_fullsized":"2024-03-31T15:56:26.023782+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049029+0000","last_change":"2024-03-31T15:56:26.041995+0000","last_active":"2024-03-31T15:56:28.049029+0000","last_peered":"2024-03-31T15:56:28.049029+0000","last_clean":"2024-03-31T15:56:28.049029+0000","last_became_active":"2024-03-31T15:56:26.041698+0000","last_became_peered":"2024-03-31T15:56:26.041698+0000","last_unstale":"2024-03-31T15:56:28.049029+0000","last_undegraded":"2024-03-31T15:56:28.049029+0000","last_fullsized":"2024-03-31T15:56:28.049029+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:03.382018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049076+0000","last_change":"2024-03-31T15:56:27.032991+0000","last_active":"2024-03-31T15:56:28.049076+0000","last_peered":"2024-03-31T15:56:28.049076+0000","last_clean":"2024-03-31T15:56:28.049076+0000","last_became_active":"2024-03-31T15:56:27.032057+0000","last_became_peered":"2024-03-31T15:56:27.032057+0000","last_unstale":"2024-03-31T15:56:28.049076+0000","last_undegraded":"2024-03-31T15:56:28.049076+0000","last_fullsized":"2024-03-31T15:56:28.049076+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734295+0000","last_change":"2024-03-31T15:56:26.034420+0000","last_active":"2024-03-31T15:56:28.734295+0000","last_peered":"2024-03-31T15:56:28.734295+0000","last_clean":"2024-03-31T15:56:28.734295+0000","last_became_active":"2024-03-31T15:56:26.034145+0000","last_became_peered":"2024-03-31T15:56:26.034145+0000","last_unstale":"2024-03-31T15:56:28.734295+0000","last_undegraded":"2024-03-31T15:56:28.734295+0000","last_fullsized":"2024-03-31T15:56:28.734295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:18.044685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023933+0000","last_change":"2024-03-31T15:56:26.023933+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023933+0000","last_undegraded":"2024-03-31T15:56:26.023933+0000","last_fullsized":"2024-03-31T15:56:26.023933+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048878+0000","last_change":"2024-03-31T15:56:26.044785+0000","last_active":"2024-03-31T15:56:28.048878+0000","last_peered":"2024-03-31T15:56:28.048878+0000","last_clean":"2024-03-31T15:56:28.048878+0000","last_became_active":"2024-03-31T15:56:26.044515+0000","last_became_peered":"2024-03-31T15:56:26.044515+0000","last_unstale":"2024-03-31T15:56:28.048878+0000","last_undegraded":"2024-03-31T15:56:28.048878+0000","last_fullsized":"2024-03-31T15:56:28.048878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:17.779102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048748+0000","last_change":"2024-03-31T15:56:27.032521+0000","last_active":"2024-03-31T15:56:28.048748+0000","last_peered":"2024-03-31T15:56:28.048748+0000","last_clean":"2024-03-31T15:56:28.048748+0000","last_became_active":"2024-03-31T15:56:27.032235+0000","last_became_peered":"2024-03-31T15:56:27.032235+0000","last_unstale":"2024-03-31T15:56:28.048748+0000","last_undegraded":"2024-03-31T15:56:28.048748+0000","last_fullsized":"2024-03-31T15:56:28.048748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045554+0000","last_change":"2024-03-31T15:56:26.041173+0000","last_active":"2024-03-31T15:56:26.045554+0000","last_peered":"2024-03-31T15:56:26.045554+0000","last_clean":"2024-03-31T15:56:26.045554+0000","last_became_active":"2024-03-31T15:56:26.040901+0000","last_became_peered":"2024-03-31T15:56:26.040901+0000","last_unstale":"2024-03-31T15:56:26.045554+0000","last_undegraded":"2024-03-31T15:56:26.045554+0000","last_fullsized":"2024-03-31T15:56:26.045554+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022785+0000","last_change":"2024-03-31T15:56:26.022785+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022785+0000","last_undegraded":"2024-03-31T15:56:26.022785+0000","last_fullsized":"2024-03-31T15:56:26.022785+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733993+0000","last_change":"2024-03-31T15:56:26.035713+0000","last_active":"2024-03-31T15:56:28.733993+0000","last_peered":"2024-03-31T15:56:28.733993+0000","last_clean":"2024-03-31T15:56:28.733993+0000","last_became_active":"2024-03-31T15:56:26.035393+0000","last_became_peered":"2024-03-31T15:56:26.035393+0000","last_unstale":"2024-03-31T15:56:28.733993+0000","last_undegraded":"2024-03-31T15:56:28.733993+0000","last_fullsized":"2024-03-31T15:56:28.733993+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:10.726676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048441+0000","last_change":"2024-03-31T15:56:27.041906+0000","last_active":"2024-03-31T15:56:28.048441+0000","last_peered":"2024-03-31T15:56:28.048441+0000","last_clean":"2024-03-31T15:56:28.048441+0000","last_became_active":"2024-03-31T15:56:27.041609+0000","last_became_peered":"2024-03-31T15:56:27.041609+0000","last_unstale":"2024-03-31T15:56:28.048441+0000","last_undegraded":"2024-03-31T15:56:28.048441+0000","last_fullsized":"2024-03-31T15:56:28.048441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:53:45.939898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048116+0000","last_change":"2024-03-31T15:56:26.041375+0000","last_active":"2024-03-31T15:56:26.048116+0000","last_peered":"2024-03-31T15:56:26.048116+0000","last_clean":"2024-03-31T15:56:26.048116+0000","last_became_active":"2024-03-31T15:56:26.041158+0000","last_became_peered":"2024-03-31T15:56:26.041158+0000","last_unstale":"2024-03-31T15:56:26.048116+0000","last_undegraded":"2024-03-31T15:56:26.048116+0000","last_fullsized":"2024-03-31T15:56:26.048116+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.024021+0000","last_change":"2024-03-31T15:56:26.024021+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.024021+0000","last_undegraded":"2024-03-31T15:56:26.024021+0000","last_fullsized":"2024-03-31T15:56:26.024021+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041150+0000","last_change":"2024-03-31T15:56:26.037434+0000","last_active":"2024-03-31T15:56:26.041150+0000","last_peered":"2024-03-31T15:56:26.041150+0000","last_clean":"2024-03-31T15:56:26.041150+0000","last_became_active":"2024-03-31T15:56:26.036753+0000","last_became_peered":"2024-03-31T15:56:26.036753+0000","last_unstale":"2024-03-31T15:56:26.041150+0000","last_undegraded":"2024-03-31T15:56:26.041150+0000","last_fullsized":"2024-03-31T15:56:26.041150+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.472784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047988+0000","last_change":"2024-03-31T15:56:27.037112+0000","last_active":"2024-03-31T15:56:28.047988+0000","last_peered":"2024-03-31T15:56:28.047988+0000","last_clean":"2024-03-31T15:56:28.047988+0000","last_became_active":"2024-03-31T15:56:27.036824+0000","last_became_peered":"2024-03-31T15:56:27.036824+0000","last_unstale":"2024-03-31T15:56:28.047988+0000","last_undegraded":"2024-03-31T15:56:28.047988+0000","last_fullsized":"2024-03-31T15:56:28.047988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:44:17.091367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047288+0000","last_change":"2024-03-31T15:56:26.038121+0000","last_active":"2024-03-31T15:56:28.047288+0000","last_peered":"2024-03-31T15:56:28.047288+0000","last_clean":"2024-03-31T15:56:28.047288+0000","last_became_active":"2024-03-31T15:56:26.037834+0000","last_became_peered":"2024-03-31T15:56:26.037834+0000","last_unstale":"2024-03-31T15:56:28.047288+0000","last_undegraded":"2024-03-31T15:56:28.047288+0000","last_fullsized":"2024-03-31T15:56:28.047288+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:38.551620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048667+0000","last_change":"2024-03-31T15:56:27.038762+0000","last_active":"2024-03-31T15:56:28.048667+0000","last_peered":"2024-03-31T15:56:28.048667+0000","last_clean":"2024-03-31T15:56:28.048667+0000","last_became_active":"2024-03-31T15:56:27.038442+0000","last_became_peered":"2024-03-31T15:56:27.038442+0000","last_unstale":"2024-03-31T15:56:28.048667+0000","last_undegraded":"2024-03-31T15:56:28.048667+0000","last_fullsized":"2024-03-31T15:56:28.048667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:23.516599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734348+0000","last_change":"2024-03-31T15:56:26.034027+0000","last_active":"2024-03-31T15:56:28.734348+0000","last_peered":"2024-03-31T15:56:28.734348+0000","last_clean":"2024-03-31T15:56:28.734348+0000","last_became_active":"2024-03-31T15:56:26.033735+0000","last_became_peered":"2024-03-31T15:56:26.033735+0000","last_unstale":"2024-03-31T15:56:28.734348+0000","last_undegraded":"2024-03-31T15:56:28.734348+0000","last_fullsized":"2024-03-31T15:56:28.734348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:46.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027856+0000","last_change":"2024-03-31T15:56:26.027856+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027856+0000","last_undegraded":"2024-03-31T15:56:26.027856+0000","last_fullsized":"2024-03-31T15:56:26.027856+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042183+0000","last_change":"2024-03-31T15:56:26.039370+0000","last_active":"2024-03-31T15:56:26.042183+0000","last_peered":"2024-03-31T15:56:26.042183+0000","last_clean":"2024-03-31T15:56:26.042183+0000","last_became_active":"2024-03-31T15:56:26.038769+0000","last_became_peered":"2024-03-31T15:56:26.038769+0000","last_unstale":"2024-03-31T15:56:26.042183+0000","last_undegraded":"2024-03-31T15:56:26.042183+0000","last_fullsized":"2024-03-31T15:56:26.042183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:15:36.882522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734424+0000","last_change":"2024-03-31T15:56:27.040609+0000","last_active":"2024-03-31T15:56:28.734424+0000","last_peered":"2024-03-31T15:56:28.734424+0000","last_clean":"2024-03-31T15:56:28.734424+0000","last_became_active":"2024-03-31T15:56:27.040329+0000","last_became_peered":"2024-03-31T15:56:27.040329+0000","last_unstale":"2024-03-31T15:56:28.734424+0000","last_undegraded":"2024-03-31T15:56:28.734424+0000","last_fullsized":"2024-03-31T15:56:28.734424+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:25.079313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041058+0000","last_change":"2024-03-31T15:56:26.036191+0000","last_active":"2024-03-31T15:56:26.041058+0000","last_peered":"2024-03-31T15:56:26.041058+0000","last_clean":"2024-03-31T15:56:26.041058+0000","last_became_active":"2024-03-31T15:56:26.035873+0000","last_became_peered":"2024-03-31T15:56:26.035873+0000","last_unstale":"2024-03-31T15:56:26.041058+0000","last_undegraded":"2024-03-31T15:56:26.041058+0000","last_fullsized":"2024-03-31T15:56:26.041058+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048941+0000","last_change":"2024-03-31T15:56:27.034376+0000","last_active":"2024-03-31T15:56:28.048941+0000","last_peered":"2024-03-31T15:56:28.048941+0000","last_clean":"2024-03-31T15:56:28.048941+0000","last_became_active":"2024-03-31T15:56:27.032545+0000","last_became_peered":"2024-03-31T15:56:27.032545+0000","last_unstale":"2024-03-31T15:56:28.048941+0000","last_undegraded":"2024-03-31T15:56:28.048941+0000","last_fullsized":"2024-03-31T15:56:28.048941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045774+0000","last_change":"2024-03-31T15:56:26.044352+0000","last_active":"2024-03-31T15:56:26.045774+0000","last_peered":"2024-03-31T15:56:26.045774+0000","last_clean":"2024-03-31T15:56:26.045774+0000","last_became_active":"2024-03-31T15:56:26.044071+0000","last_became_peered":"2024-03-31T15:56:26.044071+0000","last_unstale":"2024-03-31T15:56:26.045774+0000","last_undegraded":"2024-03-31T15:56:26.045774+0000","last_fullsized":"2024-03-31T15:56:26.045774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022357+0000","last_change":"2024-03-31T15:56:26.022357+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022357+0000","last_undegraded":"2024-03-31T15:56:26.022357+0000","last_fullsized":"2024-03-31T15:56:26.022357+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734169+0000","last_change":"2024-03-31T15:56:26.030253+0000","last_active":"2024-03-31T15:56:28.734169+0000","last_peered":"2024-03-31T15:56:28.734169+0000","last_clean":"2024-03-31T15:56:28.734169+0000","last_became_active":"2024-03-31T15:56:26.029995+0000","last_became_peered":"2024-03-31T15:56:26.029995+0000","last_unstale":"2024-03-31T15:56:28.734169+0000","last_undegraded":"2024-03-31T15:56:28.734169+0000","last_fullsized":"2024-03-31T15:56:28.734169+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:45:21.317263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025805+0000","last_change":"2024-03-31T15:56:26.025805+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025805+0000","last_undegraded":"2024-03-31T15:56:26.025805+0000","last_fullsized":"2024-03-31T15:56:26.025805+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041491+0000","last_change":"2024-03-31T15:56:26.037979+0000","last_active":"2024-03-31T15:56:26.041491+0000","last_peered":"2024-03-31T15:56:26.041491+0000","last_clean":"2024-03-31T15:56:26.041491+0000","last_became_active":"2024-03-31T15:56:26.037707+0000","last_became_peered":"2024-03-31T15:56:26.037707+0000","last_unstale":"2024-03-31T15:56:26.041491+0000","last_undegraded":"2024-03-31T15:56:26.041491+0000","last_fullsized":"2024-03-31T15:56:26.041491+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.026702+0000","last_change":"2024-03-31T15:56:26.026702+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.026702+0000","last_undegraded":"2024-03-31T15:56:26.026702+0000","last_fullsized":"2024-03-31T15:56:26.026702+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.036608+0000","last_change":"2024-03-31T15:56:26.030797+0000","last_active":"2024-03-31T15:56:26.036608+0000","last_peered":"2024-03-31T15:56:26.036608+0000","last_clean":"2024-03-31T15:56:26.036608+0000","last_became_active":"2024-03-31T15:56:26.030470+0000","last_became_peered":"2024-03-31T15:56:26.030470+0000","last_unstale":"2024-03-31T15:56:26.036608+0000","last_undegraded":"2024-03-31T15:56:26.036608+0000","last_fullsized":"2024-03-31T15:56:26.036608+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:49:10.369692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.025774+0000","last_change":"2024-03-31T15:56:26.025774+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.025774+0000","last_undegraded":"2024-03-31T15:56:26.025774+0000","last_fullsized":"2024-03-31T15:56:26.025774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041438+0000","last_change":"2024-03-31T15:56:26.036552+0000","last_active":"2024-03-31T15:56:26.041438+0000","last_peered":"2024-03-31T15:56:26.041438+0000","last_clean":"2024-03-31T15:56:26.041438+0000","last_became_active":"2024-03-31T15:56:26.036319+0000","last_became_peered":"2024-03-31T15:56:26.036319+0000","last_unstale":"2024-03-31T15:56:26.041438+0000","last_undegraded":"2024-03-31T15:56:26.041438+0000","last_fullsized":"2024-03-31T15:56:26.041438+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:28:54.045568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.021890+0000","last_change":"2024-03-31T15:56:26.021890+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.021890+0000","last_undegraded":"2024-03-31T15:56:26.021890+0000","last_fullsized":"2024-03-31T15:56:26.021890+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.042115+0000","last_change":"2024-03-31T15:56:26.039110+0000","last_active":"2024-03-31T15:56:26.042115+0000","last_peered":"2024-03-31T15:56:26.042115+0000","last_clean":"2024-03-31T15:56:26.042115+0000","last_became_active":"2024-03-31T15:56:26.037645+0000","last_became_peered":"2024-03-31T15:56:26.037645+0000","last_unstale":"2024-03-31T15:56:26.042115+0000","last_undegraded":"2024-03-31T15:56:26.042115+0000","last_fullsized":"2024-03-31T15:56:26.042115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:37:11.711750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.023537+0000","last_change":"2024-03-31T15:56:26.023537+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.023537+0000","last_undegraded":"2024-03-31T15:56:26.023537+0000","last_fullsized":"2024-03-31T15:56:26.023537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.048112+0000","last_change":"2024-03-31T15:56:26.040713+0000","last_active":"2024-03-31T15:56:26.048112+0000","last_peered":"2024-03-31T15:56:26.048112+0000","last_clean":"2024-03-31T15:56:26.048112+0000","last_became_active":"2024-03-31T15:56:26.040391+0000","last_became_peered":"2024-03-31T15:56:26.040391+0000","last_unstale":"2024-03-31T15:56:26.048112+0000","last_undegraded":"2024-03-31T15:56:26.048112+0000","last_fullsized":"2024-03-31T15:56:26.048112+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.027172+0000","last_change":"2024-03-31T15:56:26.027172+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.027172+0000","last_undegraded":"2024-03-31T15:56:26.027172+0000","last_fullsized":"2024-03-31T15:56:26.027172+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041125+0000","last_change":"2024-03-31T15:56:26.035712+0000","last_active":"2024-03-31T15:56:26.041125+0000","last_peered":"2024-03-31T15:56:26.041125+0000","last_clean":"2024-03-31T15:56:26.041125+0000","last_became_active":"2024-03-31T15:56:26.034040+0000","last_became_peered":"2024-03-31T15:56:26.034040+0000","last_unstale":"2024-03-31T15:56:26.041125+0000","last_undegraded":"2024-03-31T15:56:26.041125+0000","last_fullsized":"2024-03-31T15:56:26.041125+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:43.852072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.039713+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.037623+0000","last_became_peered":"2024-03-31T15:56:27.037623+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:10.759953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.041511+0000","last_change":"2024-03-31T15:56:26.036984+0000","last_active":"2024-03-31T15:56:26.041511+0000","last_peered":"2024-03-31T15:56:26.041511+0000","last_clean":"2024-03-31T15:56:26.041511+0000","last_became_active":"2024-03-31T15:56:26.036674+0000","last_became_peered":"2024-03-31T15:56:26.036674+0000","last_unstale":"2024-03-31T15:56:26.041511+0000","last_undegraded":"2024-03-31T15:56:26.041511+0000","last_fullsized":"2024-03-31T15:56:26.041511+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:07:56.779167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048608+0000","last_change":"2024-03-31T15:56:27.037261+0000","last_active":"2024-03-31T15:56:28.048608+0000","last_peered":"2024-03-31T15:56:28.048608+0000","last_clean":"2024-03-31T15:56:28.048608+0000","last_became_active":"2024-03-31T15:56:27.036282+0000","last_became_peered":"2024-03-31T15:56:27.036282+0000","last_unstale":"2024-03-31T15:56:28.048608+0000","last_undegraded":"2024-03-31T15:56:28.048608+0000","last_fullsized":"2024-03-31T15:56:28.048608+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:25.749230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.047955+0000","last_change":"2024-03-31T15:56:26.042322+0000","last_active":"2024-03-31T15:56:26.047955+0000","last_peered":"2024-03-31T15:56:26.047955+0000","last_clean":"2024-03-31T15:56:26.047955+0000","last_became_active":"2024-03-31T15:56:26.041942+0000","last_became_peered":"2024-03-31T15:56:26.041942+0000","last_unstale":"2024-03-31T15:56:26.047955+0000","last_undegraded":"2024-03-31T15:56:26.047955+0000","last_fullsized":"2024-03-31T15:56:26.047955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-03-31T15:56:26.022096+0000","last_change":"2024-03-31T15:56:26.022096+0000","last_active":"2024-03-31T15:56:26.009477+0000","last_peered":"2024-03-31T15:56:26.009477+0000","last_clean":"2024-03-31T15:56:26.009477+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-31T15:56:26.022096+0000","last_undegraded":"2024-03-31T15:56:26.022096+0000","last_fullsized":"2024-03-31T15:56:26.022096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.043837+0000","last_change":"2024-03-31T15:56:26.041462+0000","last_active":"2024-03-31T15:56:26.043837+0000","last_peered":"2024-03-31T15:56:26.043837+0000","last_clean":"2024-03-31T15:56:26.043837+0000","last_became_active":"2024-03-31T15:56:26.041184+0000","last_became_peered":"2024-03-31T15:56:26.041184+0000","last_unstale":"2024-03-31T15:56:26.043837+0000","last_undegraded":"2024-03-31T15:56:26.043837+0000","last_fullsized":"2024-03-31T15:56:26.043837+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:07:30.039105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.042943+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.042359+0000","last_became_peered":"2024-03-31T15:56:27.042359+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:00.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733822+0000","last_change":"2024-03-31T15:56:26.036968+0000","last_active":"2024-03-31T15:56:28.733822+0000","last_peered":"2024-03-31T15:56:28.733822+0000","last_clean":"2024-03-31T15:56:28.733822+0000","last_became_active":"2024-03-31T15:56:26.036405+0000","last_became_peered":"2024-03-31T15:56:26.036405+0000","last_unstale":"2024-03-31T15:56:28.733822+0000","last_undegraded":"2024-03-31T15:56:28.733822+0000","last_fullsized":"2024-03-31T15:56:28.733822+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:28:07.298713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733795+0000","last_change":"2024-03-31T15:56:27.057654+0000","last_active":"2024-03-31T15:56:28.733795+0000","last_peered":"2024-03-31T15:56:28.733795+0000","last_clean":"2024-03-31T15:56:28.733795+0000","last_became_active":"2024-03-31T15:56:27.057124+0000","last_became_peered":"2024-03-31T15:56:27.057124+0000","last_unstale":"2024-03-31T15:56:28.733795+0000","last_undegraded":"2024-03-31T15:56:28.733795+0000","last_fullsized":"2024-03-31T15:56:28.733795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-31T15:56:26.045156+0000","last_change":"2024-03-31T15:56:26.039648+0000","last_active":"2024-03-31T15:56:26.045156+0000","last_peered":"2024-03-31T15:56:26.045156+0000","last_clean":"2024-03-31T15:56:26.045156+0000","last_became_active":"2024-03-31T15:56:26.039276+0000","last_became_peered":"2024-03-31T15:56:26.039276+0000","last_unstale":"2024-03-31T15:56:26.045156+0000","last_undegraded":"2024-03-31T15:56:26.045156+0000","last_fullsized":"2024-03-31T15:56:26.045156+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:18:41.206663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047349+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:28.047349+0000","last_peered":"2024-03-31T15:56:28.047349+0000","last_clean":"2024-03-31T15:56:28.047349+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:28.047349+0000","last_undegraded":"2024-03-31T15:56:28.047349+0000","last_fullsized":"2024-03-31T15:56:28.047349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":146,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":273822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509478,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":280481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":870849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1253376,"data_stored":884868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:56:35.766 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:56:35.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.908+0000 7f8211a00640 1 Processor -- start 2024-03-31T15:56:35.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.908+0000 7f8211a00640 1 -- start start 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8211a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f820c15c120 0x7f820c15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8211a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c0d2b10 con 0x7f820c0d2680 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8211a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c160350 con 0x7f820c0cca30 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8211a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f820c160bf0 con 0x7f820c15c120 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820aa00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f820c15c120 0x7f820c15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8203e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f820c0d2680 0x7f820c0b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38168/0 (socket says 172.21.15.90:38168) 2024-03-31T15:56:35.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f8203e00640 1 -- 172.21.15.90:0/2997613728 learned_addr learned my addr 172.21.15.90:0/2997613728 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:35.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820a000640 1 -- 172.21.15.90:0/2997613728 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524257616 0 0) 0x7f820c0d2b10 con 0x7f820c0d2680 2024-03-31T15:56:35.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820a000640 1 -- 172.21.15.90:0/2997613728 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81f0003530 con 0x7f820c0d2680 2024-03-31T15:56:35.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820aa00640 1 -- 172.21.15.90:0/2997613728 >> v1:172.21.15.102:6789/0 conn(0x7f820c0cca30 legacy=0x7f820c0d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:35.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820aa00640 1 -- 172.21.15.90:0/2997613728 >> v1:172.21.15.90:6789/0 conn(0x7f820c0d2680 legacy=0x7f820c0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:35.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.912+0000 7f820aa00640 1 -- 172.21.15.90:0/2997613728 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c160630 con 0x7f820c15c120 2024-03-31T15:56:35.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f820aa00640 1 --2- 172.21.15.90:0/2997613728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c15c120 0x7f820c15fd00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097c0 tx=0x7f81fc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fc121b0621b1f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:35.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f820a000640 1 -- 172.21.15.90:0/2997613728 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0b1070 con 0x7f820c15c120 2024-03-31T15:56:35.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f820a000640 1 -- 172.21.15.90:0/2997613728 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81fc0a8050 con 0x7f820c15c120 2024-03-31T15:56:35.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f820a000640 1 -- 172.21.15.90:0/2997613728 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0ac860 con 0x7f820c15c120 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 -- 172.21.15.90:0/2997613728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c15c120 msgr2=0x7f820c15fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 --2- 172.21.15.90:0/2997613728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c15c120 0x7f820c15fd00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0097c0 tx=0x7f81fc01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 -- 172.21.15.90:0/2997613728 shutdown_connections 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 --2- 172.21.15.90:0/2997613728 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c15c120 0x7f820c15fd00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 -- 172.21.15.90:0/2997613728 >> 172.21.15.90:0/2997613728 conn(0x7f820c09aa40 msgr2=0x7f820c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 -- 172.21.15.90:0/2997613728 shutdown_connections 2024-03-31T15:56:35.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.916+0000 7f8211a00640 1 -- 172.21.15.90:0/2997613728 wait complete. 2024-03-31T15:56:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 Processor -- start 2024-03-31T15:56:35.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- start start 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 0x7f820c167af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f820c15c120 0x7f820c158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0b2f70 con 0x7f820c0d2680 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0b2df0 con 0x7f820c15c120 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f820c0d0bf0 con 0x7f820c0cca30 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59592/0 (socket says 172.21.15.90:59592) 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 -- 172.21.15.90:0/2589095363 learned_addr learned my addr 172.21.15.90:0/2589095363 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f820b400640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f820c15c120 0x7f820c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:35.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f820aa00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 0x7f820c167af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 -- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 msgr2=0x7f820c167af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 0x7f820c167af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 -- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f820c15c120 msgr2=0x7f820c158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f820c15c120 0x7f820c158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 -- 172.21.15.90:0/2589095363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c15b370 con 0x7f820c0d2680 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f820aa00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 0x7f820c167af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:35.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8203e00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c4d0 tx=0x7f81f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4017070 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f4005cf0 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4007270 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c0aef90 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c0af550 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.920+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820c0686d0 con 0x7f820c0d2680 2024-03-31T15:56:35.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.924+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f81f401a070 con 0x7f820c0d2680 2024-03-31T15:56:35.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.924+0000 7f8202a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 0x7f81e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:35.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.924+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f81f4085f20 con 0x7f820c0d2680 2024-03-31T15:56:35.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.924+0000 7f820aa00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 0x7f81e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:35.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.924+0000 7f820aa00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 0x7f81e806a840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0ae000 tx=0x7f81fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:35.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:35.932+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f81f4055c60 con 0x7f820c0d2680 2024-03-31T15:56:36.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.088+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f820c0c5190 con 0x7f81e8068380 2024-03-31T15:56:36.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.096+0000 7f8202a00640 1 -- 172.21.15.90:0/2589095363 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f820c0c5190 con 0x7f81e8068380 2024-03-31T15:56:36.091 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:36.092 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 msgr2=0x7f81e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 0x7f81e806a840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0ae000 tx=0x7f81fc008040 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 msgr2=0x7f820c15aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c4d0 tx=0x7f81f400c990 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 shutdown_connections 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f81e8068380 0x7f81e806a840 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f820c15c120 0x7f820c158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f820c0d2680 0x7f820c15aa90 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 --2- 172.21.15.90:0/2589095363 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f820c0cca30 0x7f820c167af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 >> 172.21.15.90:0/2589095363 conn(0x7f820c09aa40 msgr2=0x7f820c0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 shutdown_connections 2024-03-31T15:56:36.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.100+0000 7f8211a00640 1 -- 172.21.15.90:0/2589095363 wait complete. 2024-03-31T15:56:36.117 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-03-31T15:56:35.256264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222512,"kb_used_data":6312,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752208,"statfs":{"total":773094113280,"available":772866260992,"internally_reserved":0,"allocated":6463488,"data_stored":3540276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.236540"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732549+0000","last_change":"2024-03-31T15:56:27.060874+0000","last_active":"2024-03-31T15:56:28.732549+0000","last_peered":"2024-03-31T15:56:28.732549+0000","last_clean":"2024-03-31T15:56:28.732549+0000","last_became_active":"2024-03-31T15:56:27.060630+0000","last_became_peered":"2024-03-31T15:56:27.060630+0000","last_unstale":"2024-03-31T15:56:28.732549+0000","last_undegraded":"2024-03-31T15:56:28.732549+0000","last_fullsized":"2024-03-31T15:56:28.732549+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:56:54.783818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734232+0000","last_change":"2024-03-31T15:56:27.040029+0000","last_active":"2024-03-31T15:56:28.734232+0000","last_peered":"2024-03-31T15:56:28.734232+0000","last_clean":"2024-03-31T15:56:28.734232+0000","last_became_active":"2024-03-31T15:56:27.036544+0000","last_became_peered":"2024-03-31T15:56:27.036544+0000","last_unstale":"2024-03-31T15:56:28.734232+0000","last_undegraded":"2024-03-31T15:56:28.734232+0000","last_fullsized":"2024-03-31T15:56:28.734232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:16:57.935165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048573+0000","last_change":"2024-03-31T15:56:27.040254+0000","last_active":"2024-03-31T15:56:28.048573+0000","last_peered":"2024-03-31T15:56:28.048573+0000","last_clean":"2024-03-31T15:56:28.048573+0000","last_became_active":"2024-03-31T15:56:27.039839+0000","last_became_peered":"2024-03-31T15:56:27.039839+0000","last_unstale":"2024-03-31T15:56:28.048573+0000","last_undegraded":"2024-03-31T15:56:28.048573+0000","last_fullsized":"2024-03-31T15:56:28.048573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:55:38.034420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047517+0000","last_change":"2024-03-31T15:56:27.051779+0000","last_active":"2024-03-31T15:56:28.047517+0000","last_peered":"2024-03-31T15:56:28.047517+0000","last_clean":"2024-03-31T15:56:28.047517+0000","last_became_active":"2024-03-31T15:56:27.051514+0000","last_became_peered":"2024-03-31T15:56:27.051514+0000","last_unstale":"2024-03-31T15:56:28.047517+0000","last_undegraded":"2024-03-31T15:56:28.047517+0000","last_fullsized":"2024-03-31T15:56:28.047517+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:10:28.264724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731537+0000","last_change":"2024-03-31T15:56:27.057953+0000","last_active":"2024-03-31T15:56:28.731537+0000","last_peered":"2024-03-31T15:56:28.731537+0000","last_clean":"2024-03-31T15:56:28.731537+0000","last_became_active":"2024-03-31T15:56:27.057680+0000","last_became_peered":"2024-03-31T15:56:27.057680+0000","last_unstale":"2024-03-31T15:56:28.731537+0000","last_undegraded":"2024-03-31T15:56:28.731537+0000","last_fullsized":"2024-03-31T15:56:28.731537+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:41:20.883103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731153+0000","last_change":"2024-03-31T15:56:27.057475+0000","last_active":"2024-03-31T15:56:28.731153+0000","last_peered":"2024-03-31T15:56:28.731153+0000","last_clean":"2024-03-31T15:56:28.731153+0000","last_became_active":"2024-03-31T15:56:27.057203+0000","last_became_peered":"2024-03-31T15:56:27.057203+0000","last_unstale":"2024-03-31T15:56:28.731153+0000","last_undegraded":"2024-03-31T15:56:28.731153+0000","last_fullsized":"2024-03-31T15:56:28.731153+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:12.640428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045634+0000","last_change":"2024-03-31T15:56:27.035718+0000","last_active":"2024-03-31T15:56:28.045634+0000","last_peered":"2024-03-31T15:56:28.045634+0000","last_clean":"2024-03-31T15:56:28.045634+0000","last_became_active":"2024-03-31T15:56:27.035408+0000","last_became_peered":"2024-03-31T15:56:27.035408+0000","last_unstale":"2024-03-31T15:56:28.045634+0000","last_undegraded":"2024-03-31T15:56:28.045634+0000","last_fullsized":"2024-03-31T15:56:28.045634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:53.267490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733707+0000","last_change":"2024-03-31T15:56:27.040713+0000","last_active":"2024-03-31T15:56:28.733707+0000","last_peered":"2024-03-31T15:56:28.733707+0000","last_clean":"2024-03-31T15:56:28.733707+0000","last_became_active":"2024-03-31T15:56:27.040439+0000","last_became_peered":"2024-03-31T15:56:27.040439+0000","last_unstale":"2024-03-31T15:56:28.733707+0000","last_undegraded":"2024-03-31T15:56:28.733707+0000","last_fullsized":"2024-03-31T15:56:28.733707+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.046355+0000","last_change":"2024-03-31T15:56:27.044656+0000","last_active":"2024-03-31T15:56:28.046355+0000","last_peered":"2024-03-31T15:56:28.046355+0000","last_clean":"2024-03-31T15:56:28.046355+0000","last_became_active":"2024-03-31T15:56:27.044334+0000","last_became_peered":"2024-03-31T15:56:27.044334+0000","last_unstale":"2024-03-31T15:56:28.046355+0000","last_undegraded":"2024-03-31T15:56:28.046355+0000","last_fullsized":"2024-03-31T15:56:28.046355+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:51.293459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731988+0000","last_change":"2024-03-31T15:56:27.039429+0000","last_active":"2024-03-31T15:56:28.731988+0000","last_peered":"2024-03-31T15:56:28.731988+0000","last_clean":"2024-03-31T15:56:28.731988+0000","last_became_active":"2024-03-31T15:56:27.039127+0000","last_became_peered":"2024-03-31T15:56:27.039127+0000","last_unstale":"2024-03-31T15:56:28.731988+0000","last_undegraded":"2024-03-31T15:56:28.731988+0000","last_fullsized":"2024-03-31T15:56:28.731988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:24.822938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731993+0000","last_change":"2024-03-31T15:56:27.037875+0000","last_active":"2024-03-31T15:56:28.731993+0000","last_peered":"2024-03-31T15:56:28.731993+0000","last_clean":"2024-03-31T15:56:28.731993+0000","last_became_active":"2024-03-31T15:56:27.037624+0000","last_became_peered":"2024-03-31T15:56:27.037624+0000","last_unstale":"2024-03-31T15:56:28.731993+0000","last_undegraded":"2024-03-31T15:56:28.731993+0000","last_fullsized":"2024-03-31T15:56:28.731993+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:51.293459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047689+0000","last_change":"2024-03-31T15:56:27.030666+0000","last_active":"2024-03-31T15:56:28.047689+0000","last_peered":"2024-03-31T15:56:28.047689+0000","last_clean":"2024-03-31T15:56:28.047689+0000","last_became_active":"2024-03-31T15:56:27.029271+0000","last_became_peered":"2024-03-31T15:56:27.029271+0000","last_unstale":"2024-03-31T15:56:28.047689+0000","last_undegraded":"2024-03-31T15:56:28.047689+0000","last_fullsized":"2024-03-31T15:56:28.047689+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:08:17.672951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231278+0000","last_change":"2024-03-31T15:56:27.051139+0000","last_active":"2024-03-31T15:56:28.231278+0000","last_peered":"2024-03-31T15:56:28.231278+0000","last_clean":"2024-03-31T15:56:28.231278+0000","last_became_active":"2024-03-31T15:56:27.050855+0000","last_became_peered":"2024-03-31T15:56:27.050855+0000","last_unstale":"2024-03-31T15:56:28.231278+0000","last_undegraded":"2024-03-31T15:56:28.231278+0000","last_fullsized":"2024-03-31T15:56:28.231278+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:39.134285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045936+0000","last_change":"2024-03-31T15:56:27.057877+0000","last_active":"2024-03-31T15:56:28.045936+0000","last_peered":"2024-03-31T15:56:28.045936+0000","last_clean":"2024-03-31T15:56:28.045936+0000","last_became_active":"2024-03-31T15:56:27.057574+0000","last_became_peered":"2024-03-31T15:56:27.057574+0000","last_unstale":"2024-03-31T15:56:28.045936+0000","last_undegraded":"2024-03-31T15:56:28.045936+0000","last_fullsized":"2024-03-31T15:56:28.045936+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:24.822938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045798+0000","last_change":"2024-03-31T15:56:27.038040+0000","last_active":"2024-03-31T15:56:28.045798+0000","last_peered":"2024-03-31T15:56:28.045798+0000","last_clean":"2024-03-31T15:56:28.045798+0000","last_became_active":"2024-03-31T15:56:27.037754+0000","last_became_peered":"2024-03-31T15:56:27.037754+0000","last_unstale":"2024-03-31T15:56:28.045798+0000","last_undegraded":"2024-03-31T15:56:28.045798+0000","last_fullsized":"2024-03-31T15:56:28.045798+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:24.475678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731498+0000","last_change":"2024-03-31T15:56:27.043183+0000","last_active":"2024-03-31T15:56:28.731498+0000","last_peered":"2024-03-31T15:56:28.731498+0000","last_clean":"2024-03-31T15:56:28.731498+0000","last_became_active":"2024-03-31T15:56:27.042941+0000","last_became_peered":"2024-03-31T15:56:27.042941+0000","last_unstale":"2024-03-31T15:56:28.731498+0000","last_undegraded":"2024-03-31T15:56:28.731498+0000","last_fullsized":"2024-03-31T15:56:28.731498+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:09.450355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734398+0000","last_change":"2024-03-31T15:56:27.041513+0000","last_active":"2024-03-31T15:56:28.734398+0000","last_peered":"2024-03-31T15:56:28.734398+0000","last_clean":"2024-03-31T15:56:28.734398+0000","last_became_active":"2024-03-31T15:56:27.041210+0000","last_became_peered":"2024-03-31T15:56:27.041210+0000","last_unstale":"2024-03-31T15:56:28.734398+0000","last_undegraded":"2024-03-31T15:56:28.734398+0000","last_fullsized":"2024-03-31T15:56:28.734398+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:12:55.146299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048591+0000","last_change":"2024-03-31T15:56:27.049282+0000","last_active":"2024-03-31T15:56:28.048591+0000","last_peered":"2024-03-31T15:56:28.048591+0000","last_clean":"2024-03-31T15:56:28.048591+0000","last_became_active":"2024-03-31T15:56:27.048536+0000","last_became_peered":"2024-03-31T15:56:27.048536+0000","last_unstale":"2024-03-31T15:56:28.048591+0000","last_undegraded":"2024-03-31T15:56:28.048591+0000","last_fullsized":"2024-03-31T15:56:28.048591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:49:37.697269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732485+0000","last_change":"2024-03-31T15:56:27.060292+0000","last_active":"2024-03-31T15:56:28.732485+0000","last_peered":"2024-03-31T15:56:28.732485+0000","last_clean":"2024-03-31T15:56:28.732485+0000","last_became_active":"2024-03-31T15:56:27.060044+0000","last_became_peered":"2024-03-31T15:56:27.060044+0000","last_unstale":"2024-03-31T15:56:28.732485+0000","last_undegraded":"2024-03-31T15:56:28.732485+0000","last_fullsized":"2024-03-31T15:56:28.732485+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:03.617564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734311+0000","last_change":"2024-03-31T15:56:27.059752+0000","last_active":"2024-03-31T15:56:28.734311+0000","last_peered":"2024-03-31T15:56:28.734311+0000","last_clean":"2024-03-31T15:56:28.734311+0000","last_became_active":"2024-03-31T15:56:27.059504+0000","last_became_peered":"2024-03-31T15:56:27.059504+0000","last_unstale":"2024-03-31T15:56:28.734311+0000","last_undegraded":"2024-03-31T15:56:28.734311+0000","last_fullsized":"2024-03-31T15:56:28.734311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:32:03.532453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048464+0000","last_change":"2024-03-31T15:56:27.030882+0000","last_active":"2024-03-31T15:56:28.048464+0000","last_peered":"2024-03-31T15:56:28.048464+0000","last_clean":"2024-03-31T15:56:28.048464+0000","last_became_active":"2024-03-31T15:56:27.030222+0000","last_became_peered":"2024-03-31T15:56:27.030222+0000","last_unstale":"2024-03-31T15:56:28.048464+0000","last_undegraded":"2024-03-31T15:56:28.048464+0000","last_fullsized":"2024-03-31T15:56:28.048464+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:25:25.459268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733876+0000","last_change":"2024-03-31T15:56:27.053182+0000","last_active":"2024-03-31T15:56:28.733876+0000","last_peered":"2024-03-31T15:56:28.733876+0000","last_clean":"2024-03-31T15:56:28.733876+0000","last_became_active":"2024-03-31T15:56:27.052882+0000","last_became_peered":"2024-03-31T15:56:27.052882+0000","last_unstale":"2024-03-31T15:56:28.733876+0000","last_undegraded":"2024-03-31T15:56:28.733876+0000","last_fullsized":"2024-03-31T15:56:28.733876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047955+0000","last_change":"2024-03-31T15:56:27.037326+0000","last_active":"2024-03-31T15:56:28.047955+0000","last_peered":"2024-03-31T15:56:28.047955+0000","last_clean":"2024-03-31T15:56:28.047955+0000","last_became_active":"2024-03-31T15:56:27.036842+0000","last_became_peered":"2024-03-31T15:56:27.036842+0000","last_unstale":"2024-03-31T15:56:28.047955+0000","last_undegraded":"2024-03-31T15:56:28.047955+0000","last_fullsized":"2024-03-31T15:56:28.047955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:13.092644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231136+0000","last_change":"2024-03-31T15:56:27.050086+0000","last_active":"2024-03-31T15:56:28.231136+0000","last_peered":"2024-03-31T15:56:28.231136+0000","last_clean":"2024-03-31T15:56:28.231136+0000","last_became_active":"2024-03-31T15:56:27.049803+0000","last_became_peered":"2024-03-31T15:56:27.049803+0000","last_unstale":"2024-03-31T15:56:28.231136+0000","last_undegraded":"2024-03-31T15:56:28.231136+0000","last_fullsized":"2024-03-31T15:56:28.231136+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:59:49.609647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732648+0000","last_change":"2024-03-31T15:56:27.058506+0000","last_active":"2024-03-31T15:56:28.732648+0000","last_peered":"2024-03-31T15:56:28.732648+0000","last_clean":"2024-03-31T15:56:28.732648+0000","last_became_active":"2024-03-31T15:56:27.058252+0000","last_became_peered":"2024-03-31T15:56:27.058252+0000","last_unstale":"2024-03-31T15:56:28.732648+0000","last_undegraded":"2024-03-31T15:56:28.732648+0000","last_fullsized":"2024-03-31T15:56:28.732648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:49:03.064057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047964+0000","last_change":"2024-03-31T15:56:27.042622+0000","last_active":"2024-03-31T15:56:28.047964+0000","last_peered":"2024-03-31T15:56:28.047964+0000","last_clean":"2024-03-31T15:56:28.047964+0000","last_became_active":"2024-03-31T15:56:27.041200+0000","last_became_peered":"2024-03-31T15:56:27.041200+0000","last_unstale":"2024-03-31T15:56:28.047964+0000","last_undegraded":"2024-03-31T15:56:28.047964+0000","last_fullsized":"2024-03-31T15:56:28.047964+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:35.296800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731909+0000","last_change":"2024-03-31T15:56:27.048467+0000","last_active":"2024-03-31T15:56:28.731909+0000","last_peered":"2024-03-31T15:56:28.731909+0000","last_clean":"2024-03-31T15:56:28.731909+0000","last_became_active":"2024-03-31T15:56:27.048190+0000","last_became_peered":"2024-03-31T15:56:27.048190+0000","last_unstale":"2024-03-31T15:56:28.731909+0000","last_undegraded":"2024-03-31T15:56:28.731909+0000","last_fullsized":"2024-03-31T15:56:28.731909+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:37:20.745451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732586+0000","last_change":"2024-03-31T15:56:27.058392+0000","last_active":"2024-03-31T15:56:28.732586+0000","last_peered":"2024-03-31T15:56:28.732586+0000","last_clean":"2024-03-31T15:56:28.732586+0000","last_became_active":"2024-03-31T15:56:27.058103+0000","last_became_peered":"2024-03-31T15:56:27.058103+0000","last_unstale":"2024-03-31T15:56:28.732586+0000","last_undegraded":"2024-03-31T15:56:28.732586+0000","last_fullsized":"2024-03-31T15:56:28.732586+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:02:38.875392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734201+0000","last_change":"2024-03-31T15:56:27.041576+0000","last_active":"2024-03-31T15:56:28.734201+0000","last_peered":"2024-03-31T15:56:28.734201+0000","last_clean":"2024-03-31T15:56:28.734201+0000","last_became_active":"2024-03-31T15:56:27.041327+0000","last_became_peered":"2024-03-31T15:56:27.041327+0000","last_unstale":"2024-03-31T15:56:28.734201+0000","last_undegraded":"2024-03-31T15:56:28.734201+0000","last_fullsized":"2024-03-31T15:56:28.734201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:16:39.162796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048359+0000","last_change":"2024-03-31T15:56:27.034473+0000","last_active":"2024-03-31T15:56:28.048359+0000","last_peered":"2024-03-31T15:56:28.048359+0000","last_clean":"2024-03-31T15:56:28.048359+0000","last_became_active":"2024-03-31T15:56:27.034209+0000","last_became_peered":"2024-03-31T15:56:27.034209+0000","last_unstale":"2024-03-31T15:56:28.048359+0000","last_undegraded":"2024-03-31T15:56:28.048359+0000","last_fullsized":"2024-03-31T15:56:28.048359+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:05:45.169835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731505+0000","last_change":"2024-03-31T15:56:27.058056+0000","last_active":"2024-03-31T15:56:28.731505+0000","last_peered":"2024-03-31T15:56:28.731505+0000","last_clean":"2024-03-31T15:56:28.731505+0000","last_became_active":"2024-03-31T15:56:27.057786+0000","last_became_peered":"2024-03-31T15:56:27.057786+0000","last_unstale":"2024-03-31T15:56:28.731505+0000","last_undegraded":"2024-03-31T15:56:28.731505+0000","last_fullsized":"2024-03-31T15:56:28.731505+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:04:01.295156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731293+0000","last_change":"2024-03-31T15:56:27.056264+0000","last_active":"2024-03-31T15:56:28.731293+0000","last_peered":"2024-03-31T15:56:28.731293+0000","last_clean":"2024-03-31T15:56:28.731293+0000","last_became_active":"2024-03-31T15:56:27.056025+0000","last_became_peered":"2024-03-31T15:56:27.056025+0000","last_unstale":"2024-03-31T15:56:28.731293+0000","last_undegraded":"2024-03-31T15:56:28.731293+0000","last_fullsized":"2024-03-31T15:56:28.731293+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:08:17.672951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732184+0000","last_change":"2024-03-31T15:56:27.045174+0000","last_active":"2024-03-31T15:56:28.732184+0000","last_peered":"2024-03-31T15:56:28.732184+0000","last_clean":"2024-03-31T15:56:28.732184+0000","last_became_active":"2024-03-31T15:56:27.044851+0000","last_became_peered":"2024-03-31T15:56:27.044851+0000","last_unstale":"2024-03-31T15:56:28.732184+0000","last_undegraded":"2024-03-31T15:56:28.732184+0000","last_fullsized":"2024-03-31T15:56:28.732184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:18:07.450618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":18,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731753+0000","last_change":"2024-03-31T15:56:26.043587+0000","last_active":"2024-03-31T15:56:28.731753+0000","last_peered":"2024-03-31T15:56:28.731753+0000","last_clean":"2024-03-31T15:56:28.731753+0000","last_became_active":"2024-03-31T15:56:26.043321+0000","last_became_peered":"2024-03-31T15:56:26.043321+0000","last_unstale":"2024-03-31T15:56:28.731753+0000","last_undegraded":"2024-03-31T15:56:28.731753+0000","last_fullsized":"2024-03-31T15:56:28.731753+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:23.820044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732801+0000","last_change":"2024-03-31T15:56:27.036484+0000","last_active":"2024-03-31T15:56:28.732801+0000","last_peered":"2024-03-31T15:56:28.732801+0000","last_clean":"2024-03-31T15:56:28.732801+0000","last_became_active":"2024-03-31T15:56:27.036252+0000","last_became_peered":"2024-03-31T15:56:27.036252+0000","last_unstale":"2024-03-31T15:56:28.732801+0000","last_undegraded":"2024-03-31T15:56:28.732801+0000","last_fullsized":"2024-03-31T15:56:28.732801+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:42.241002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734183+0000","last_change":"2024-03-31T15:56:26.029688+0000","last_active":"2024-03-31T15:56:28.734183+0000","last_peered":"2024-03-31T15:56:28.734183+0000","last_clean":"2024-03-31T15:56:28.734183+0000","last_became_active":"2024-03-31T15:56:26.029396+0000","last_became_peered":"2024-03-31T15:56:26.029396+0000","last_unstale":"2024-03-31T15:56:28.734183+0000","last_undegraded":"2024-03-31T15:56:28.734183+0000","last_fullsized":"2024-03-31T15:56:28.734183+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:52:46.291141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731809+0000","last_change":"2024-03-31T15:56:27.059114+0000","last_active":"2024-03-31T15:56:28.731809+0000","last_peered":"2024-03-31T15:56:28.731809+0000","last_clean":"2024-03-31T15:56:28.731809+0000","last_became_active":"2024-03-31T15:56:27.058846+0000","last_became_peered":"2024-03-31T15:56:27.058846+0000","last_unstale":"2024-03-31T15:56:28.731809+0000","last_undegraded":"2024-03-31T15:56:28.731809+0000","last_fullsized":"2024-03-31T15:56:28.731809+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:39:14.822208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732094+0000","last_change":"2024-03-31T15:56:26.044595+0000","last_active":"2024-03-31T15:56:28.732094+0000","last_peered":"2024-03-31T15:56:28.732094+0000","last_clean":"2024-03-31T15:56:28.732094+0000","last_became_active":"2024-03-31T15:56:26.044321+0000","last_became_peered":"2024-03-31T15:56:26.044321+0000","last_unstale":"2024-03-31T15:56:28.732094+0000","last_undegraded":"2024-03-31T15:56:28.732094+0000","last_fullsized":"2024-03-31T15:56:28.732094+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:23.472784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731989+0000","last_change":"2024-03-31T15:56:27.055630+0000","last_active":"2024-03-31T15:56:28.731989+0000","last_peered":"2024-03-31T15:56:28.731989+0000","last_clean":"2024-03-31T15:56:28.731989+0000","last_became_active":"2024-03-31T15:56:27.055080+0000","last_became_peered":"2024-03-31T15:56:27.055080+0000","last_unstale":"2024-03-31T15:56:28.731989+0000","last_undegraded":"2024-03-31T15:56:28.731989+0000","last_fullsized":"2024-03-31T15:56:28.731989+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:30:59.047999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047795+0000","last_change":"2024-03-31T15:56:26.035102+0000","last_active":"2024-03-31T15:56:28.047795+0000","last_peered":"2024-03-31T15:56:28.047795+0000","last_clean":"2024-03-31T15:56:28.047795+0000","last_became_active":"2024-03-31T15:56:26.034821+0000","last_became_peered":"2024-03-31T15:56:26.034821+0000","last_unstale":"2024-03-31T15:56:28.047795+0000","last_undegraded":"2024-03-31T15:56:28.047795+0000","last_fullsized":"2024-03-31T15:56:28.047795+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:23.820044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733891+0000","last_change":"2024-03-31T15:56:27.053152+0000","last_active":"2024-03-31T15:56:28.733891+0000","last_peered":"2024-03-31T15:56:28.733891+0000","last_clean":"2024-03-31T15:56:28.733891+0000","last_became_active":"2024-03-31T15:56:27.052870+0000","last_became_peered":"2024-03-31T15:56:27.052870+0000","last_unstale":"2024-03-31T15:56:28.733891+0000","last_undegraded":"2024-03-31T15:56:28.733891+0000","last_fullsized":"2024-03-31T15:56:28.733891+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:21:53.273112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045830+0000","last_change":"2024-03-31T15:56:26.035499+0000","last_active":"2024-03-31T15:56:28.045830+0000","last_peered":"2024-03-31T15:56:28.045830+0000","last_clean":"2024-03-31T15:56:28.045830+0000","last_became_active":"2024-03-31T15:56:26.035216+0000","last_became_peered":"2024-03-31T15:56:26.035216+0000","last_unstale":"2024-03-31T15:56:28.045830+0000","last_undegraded":"2024-03-31T15:56:28.045830+0000","last_fullsized":"2024-03-31T15:56:28.045830+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:25:28.079103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734108+0000","last_change":"2024-03-31T15:56:27.055018+0000","last_active":"2024-03-31T15:56:28.734108+0000","last_peered":"2024-03-31T15:56:28.734108+0000","last_clean":"2024-03-31T15:56:28.734108+0000","last_became_active":"2024-03-31T15:56:27.054765+0000","last_became_peered":"2024-03-31T15:56:27.054765+0000","last_unstale":"2024-03-31T15:56:28.734108+0000","last_undegraded":"2024-03-31T15:56:28.734108+0000","last_fullsized":"2024-03-31T15:56:28.734108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:26:49.516522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731361+0000","last_change":"2024-03-31T15:56:26.040614+0000","last_active":"2024-03-31T15:56:28.731361+0000","last_peered":"2024-03-31T15:56:28.731361+0000","last_clean":"2024-03-31T15:56:28.731361+0000","last_became_active":"2024-03-31T15:56:26.040371+0000","last_became_peered":"2024-03-31T15:56:26.040371+0000","last_unstale":"2024-03-31T15:56:28.731361+0000","last_undegraded":"2024-03-31T15:56:28.731361+0000","last_fullsized":"2024-03-31T15:56:28.731361+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:50.290565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231164+0000","last_change":"2024-03-31T15:56:27.041498+0000","last_active":"2024-03-31T15:56:28.231164+0000","last_peered":"2024-03-31T15:56:28.231164+0000","last_clean":"2024-03-31T15:56:28.231164+0000","last_became_active":"2024-03-31T15:56:27.041031+0000","last_became_peered":"2024-03-31T15:56:27.041031+0000","last_unstale":"2024-03-31T15:56:28.231164+0000","last_undegraded":"2024-03-31T15:56:28.231164+0000","last_fullsized":"2024-03-31T15:56:28.231164+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:27:44.854966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047146+0000","last_change":"2024-03-31T15:56:26.037304+0000","last_active":"2024-03-31T15:56:28.047146+0000","last_peered":"2024-03-31T15:56:28.047146+0000","last_clean":"2024-03-31T15:56:28.047146+0000","last_became_active":"2024-03-31T15:56:26.037038+0000","last_became_peered":"2024-03-31T15:56:26.037038+0000","last_unstale":"2024-03-31T15:56:28.047146+0000","last_undegraded":"2024-03-31T15:56:28.047146+0000","last_fullsized":"2024-03-31T15:56:28.047146+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:27.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733928+0000","last_change":"2024-03-31T15:56:27.043180+0000","last_active":"2024-03-31T15:56:28.733928+0000","last_peered":"2024-03-31T15:56:28.733928+0000","last_clean":"2024-03-31T15:56:28.733928+0000","last_became_active":"2024-03-31T15:56:27.042692+0000","last_became_peered":"2024-03-31T15:56:27.042692+0000","last_unstale":"2024-03-31T15:56:28.733928+0000","last_undegraded":"2024-03-31T15:56:28.733928+0000","last_fullsized":"2024-03-31T15:56:28.733928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:21:35.971485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045850+0000","last_change":"2024-03-31T15:56:26.040038+0000","last_active":"2024-03-31T15:56:28.045850+0000","last_peered":"2024-03-31T15:56:28.045850+0000","last_clean":"2024-03-31T15:56:28.045850+0000","last_became_active":"2024-03-31T15:56:26.039654+0000","last_became_peered":"2024-03-31T15:56:26.039654+0000","last_unstale":"2024-03-31T15:56:28.045850+0000","last_undegraded":"2024-03-31T15:56:28.045850+0000","last_fullsized":"2024-03-31T15:56:28.045850+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.046503+0000","last_change":"2024-03-31T15:56:27.037170+0000","last_active":"2024-03-31T15:56:28.046503+0000","last_peered":"2024-03-31T15:56:28.046503+0000","last_clean":"2024-03-31T15:56:28.046503+0000","last_became_active":"2024-03-31T15:56:27.036874+0000","last_became_peered":"2024-03-31T15:56:27.036874+0000","last_unstale":"2024-03-31T15:56:28.046503+0000","last_undegraded":"2024-03-31T15:56:28.046503+0000","last_fullsized":"2024-03-31T15:56:28.046503+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:42.241002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049029+0000","last_change":"2024-03-31T15:56:26.041995+0000","last_active":"2024-03-31T15:56:28.049029+0000","last_peered":"2024-03-31T15:56:28.049029+0000","last_clean":"2024-03-31T15:56:28.049029+0000","last_became_active":"2024-03-31T15:56:26.041698+0000","last_became_peered":"2024-03-31T15:56:26.041698+0000","last_unstale":"2024-03-31T15:56:28.049029+0000","last_undegraded":"2024-03-31T15:56:28.049029+0000","last_fullsized":"2024-03-31T15:56:28.049029+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:24:03.382018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.049076+0000","last_change":"2024-03-31T15:56:27.032991+0000","last_active":"2024-03-31T15:56:28.049076+0000","last_peered":"2024-03-31T15:56:28.049076+0000","last_clean":"2024-03-31T15:56:28.049076+0000","last_became_active":"2024-03-31T15:56:27.032057+0000","last_became_peered":"2024-03-31T15:56:27.032057+0000","last_unstale":"2024-03-31T15:56:28.049076+0000","last_undegraded":"2024-03-31T15:56:28.049076+0000","last_fullsized":"2024-03-31T15:56:28.049076+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:14:51.370440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734295+0000","last_change":"2024-03-31T15:56:26.034420+0000","last_active":"2024-03-31T15:56:28.734295+0000","last_peered":"2024-03-31T15:56:28.734295+0000","last_clean":"2024-03-31T15:56:28.734295+0000","last_became_active":"2024-03-31T15:56:26.034145+0000","last_became_peered":"2024-03-31T15:56:26.034145+0000","last_unstale":"2024-03-31T15:56:28.734295+0000","last_undegraded":"2024-03-31T15:56:28.734295+0000","last_fullsized":"2024-03-31T15:56:28.734295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:45:18.044685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731682+0000","last_change":"2024-03-31T15:56:27.037501+0000","last_active":"2024-03-31T15:56:28.731682+0000","last_peered":"2024-03-31T15:56:28.731682+0000","last_clean":"2024-03-31T15:56:28.731682+0000","last_became_active":"2024-03-31T15:56:27.037234+0000","last_became_peered":"2024-03-31T15:56:27.037234+0000","last_unstale":"2024-03-31T15:56:28.731682+0000","last_undegraded":"2024-03-31T15:56:28.731682+0000","last_fullsized":"2024-03-31T15:56:28.731682+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:53.267490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048878+0000","last_change":"2024-03-31T15:56:26.044785+0000","last_active":"2024-03-31T15:56:28.048878+0000","last_peered":"2024-03-31T15:56:28.048878+0000","last_clean":"2024-03-31T15:56:28.048878+0000","last_became_active":"2024-03-31T15:56:26.044515+0000","last_became_peered":"2024-03-31T15:56:26.044515+0000","last_unstale":"2024-03-31T15:56:28.048878+0000","last_undegraded":"2024-03-31T15:56:28.048878+0000","last_fullsized":"2024-03-31T15:56:28.048878+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:05:17.779102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048748+0000","last_change":"2024-03-31T15:56:27.032521+0000","last_active":"2024-03-31T15:56:28.048748+0000","last_peered":"2024-03-31T15:56:28.048748+0000","last_clean":"2024-03-31T15:56:28.048748+0000","last_became_active":"2024-03-31T15:56:27.032235+0000","last_became_peered":"2024-03-31T15:56:27.032235+0000","last_unstale":"2024-03-31T15:56:28.048748+0000","last_undegraded":"2024-03-31T15:56:28.048748+0000","last_fullsized":"2024-03-31T15:56:28.048748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:33:14.977365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.046120+0000","last_change":"2024-03-31T15:56:26.041173+0000","last_active":"2024-03-31T15:56:28.046120+0000","last_peered":"2024-03-31T15:56:28.046120+0000","last_clean":"2024-03-31T15:56:28.046120+0000","last_became_active":"2024-03-31T15:56:26.040901+0000","last_became_peered":"2024-03-31T15:56:26.040901+0000","last_unstale":"2024-03-31T15:56:28.046120+0000","last_undegraded":"2024-03-31T15:56:28.046120+0000","last_fullsized":"2024-03-31T15:56:28.046120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:09.232374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047203+0000","last_change":"2024-03-31T15:56:27.033614+0000","last_active":"2024-03-31T15:56:28.047203+0000","last_peered":"2024-03-31T15:56:28.047203+0000","last_clean":"2024-03-31T15:56:28.047203+0000","last_became_active":"2024-03-31T15:56:27.032965+0000","last_became_peered":"2024-03-31T15:56:27.032965+0000","last_unstale":"2024-03-31T15:56:28.047203+0000","last_undegraded":"2024-03-31T15:56:28.047203+0000","last_fullsized":"2024-03-31T15:56:28.047203+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:57:12.640428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733993+0000","last_change":"2024-03-31T15:56:26.035713+0000","last_active":"2024-03-31T15:56:28.733993+0000","last_peered":"2024-03-31T15:56:28.733993+0000","last_clean":"2024-03-31T15:56:28.733993+0000","last_became_active":"2024-03-31T15:56:26.035393+0000","last_became_peered":"2024-03-31T15:56:26.035393+0000","last_unstale":"2024-03-31T15:56:28.733993+0000","last_undegraded":"2024-03-31T15:56:28.733993+0000","last_fullsized":"2024-03-31T15:56:28.733993+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:34:10.726676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048441+0000","last_change":"2024-03-31T15:56:27.041906+0000","last_active":"2024-03-31T15:56:28.048441+0000","last_peered":"2024-03-31T15:56:28.048441+0000","last_clean":"2024-03-31T15:56:28.048441+0000","last_became_active":"2024-03-31T15:56:27.041609+0000","last_became_peered":"2024-03-31T15:56:27.041609+0000","last_unstale":"2024-03-31T15:56:28.048441+0000","last_undegraded":"2024-03-31T15:56:28.048441+0000","last_fullsized":"2024-03-31T15:56:28.048441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:53:45.939898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231040+0000","last_change":"2024-03-31T15:56:26.041375+0000","last_active":"2024-03-31T15:56:28.231040+0000","last_peered":"2024-03-31T15:56:28.231040+0000","last_clean":"2024-03-31T15:56:28.231040+0000","last_became_active":"2024-03-31T15:56:26.041158+0000","last_became_peered":"2024-03-31T15:56:26.041158+0000","last_unstale":"2024-03-31T15:56:28.231040+0000","last_undegraded":"2024-03-31T15:56:28.231040+0000","last_fullsized":"2024-03-31T15:56:28.231040+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:25:28.079103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045635+0000","last_change":"2024-03-31T15:56:27.039074+0000","last_active":"2024-03-31T15:56:28.045635+0000","last_peered":"2024-03-31T15:56:28.045635+0000","last_clean":"2024-03-31T15:56:28.045635+0000","last_became_active":"2024-03-31T15:56:27.038788+0000","last_became_peered":"2024-03-31T15:56:27.038788+0000","last_unstale":"2024-03-31T15:56:28.045635+0000","last_undegraded":"2024-03-31T15:56:28.045635+0000","last_fullsized":"2024-03-31T15:56:28.045635+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:39.134285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047252+0000","last_change":"2024-03-31T15:56:26.037434+0000","last_active":"2024-03-31T15:56:28.047252+0000","last_peered":"2024-03-31T15:56:28.047252+0000","last_clean":"2024-03-31T15:56:28.047252+0000","last_became_active":"2024-03-31T15:56:26.036753+0000","last_became_peered":"2024-03-31T15:56:26.036753+0000","last_unstale":"2024-03-31T15:56:28.047252+0000","last_undegraded":"2024-03-31T15:56:28.047252+0000","last_fullsized":"2024-03-31T15:56:28.047252+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:21:32.088617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047988+0000","last_change":"2024-03-31T15:56:27.037112+0000","last_active":"2024-03-31T15:56:28.047988+0000","last_peered":"2024-03-31T15:56:28.047988+0000","last_clean":"2024-03-31T15:56:28.047988+0000","last_became_active":"2024-03-31T15:56:27.036824+0000","last_became_peered":"2024-03-31T15:56:27.036824+0000","last_unstale":"2024-03-31T15:56:28.047988+0000","last_undegraded":"2024-03-31T15:56:28.047988+0000","last_fullsized":"2024-03-31T15:56:28.047988+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:44:17.091367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047288+0000","last_change":"2024-03-31T15:56:26.038121+0000","last_active":"2024-03-31T15:56:28.047288+0000","last_peered":"2024-03-31T15:56:28.047288+0000","last_clean":"2024-03-31T15:56:28.047288+0000","last_became_active":"2024-03-31T15:56:26.037834+0000","last_became_peered":"2024-03-31T15:56:26.037834+0000","last_unstale":"2024-03-31T15:56:28.047288+0000","last_undegraded":"2024-03-31T15:56:28.047288+0000","last_fullsized":"2024-03-31T15:56:28.047288+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:01:38.551620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048667+0000","last_change":"2024-03-31T15:56:27.038762+0000","last_active":"2024-03-31T15:56:28.048667+0000","last_peered":"2024-03-31T15:56:28.048667+0000","last_clean":"2024-03-31T15:56:28.048667+0000","last_became_active":"2024-03-31T15:56:27.038442+0000","last_became_peered":"2024-03-31T15:56:27.038442+0000","last_unstale":"2024-03-31T15:56:28.048667+0000","last_undegraded":"2024-03-31T15:56:28.048667+0000","last_fullsized":"2024-03-31T15:56:28.048667+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:19:23.516599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734348+0000","last_change":"2024-03-31T15:56:26.034027+0000","last_active":"2024-03-31T15:56:28.734348+0000","last_peered":"2024-03-31T15:56:28.734348+0000","last_clean":"2024-03-31T15:56:28.734348+0000","last_became_active":"2024-03-31T15:56:26.033735+0000","last_became_peered":"2024-03-31T15:56:26.033735+0000","last_unstale":"2024-03-31T15:56:28.734348+0000","last_undegraded":"2024-03-31T15:56:28.734348+0000","last_fullsized":"2024-03-31T15:56:28.734348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:05:46.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047565+0000","last_change":"2024-03-31T15:56:27.032281+0000","last_active":"2024-03-31T15:56:28.047565+0000","last_peered":"2024-03-31T15:56:28.047565+0000","last_clean":"2024-03-31T15:56:28.047565+0000","last_became_active":"2024-03-31T15:56:27.031944+0000","last_became_peered":"2024-03-31T15:56:27.031944+0000","last_unstale":"2024-03-31T15:56:28.047565+0000","last_undegraded":"2024-03-31T15:56:28.047565+0000","last_fullsized":"2024-03-31T15:56:28.047565+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:45:09.450355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732188+0000","last_change":"2024-03-31T15:56:26.039370+0000","last_active":"2024-03-31T15:56:28.732188+0000","last_peered":"2024-03-31T15:56:28.732188+0000","last_clean":"2024-03-31T15:56:28.732188+0000","last_became_active":"2024-03-31T15:56:26.038769+0000","last_became_peered":"2024-03-31T15:56:26.038769+0000","last_unstale":"2024-03-31T15:56:28.732188+0000","last_undegraded":"2024-03-31T15:56:28.732188+0000","last_fullsized":"2024-03-31T15:56:28.732188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:41.238108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734424+0000","last_change":"2024-03-31T15:56:27.040609+0000","last_active":"2024-03-31T15:56:28.734424+0000","last_peered":"2024-03-31T15:56:28.734424+0000","last_clean":"2024-03-31T15:56:28.734424+0000","last_became_active":"2024-03-31T15:56:27.040329+0000","last_became_peered":"2024-03-31T15:56:27.040329+0000","last_unstale":"2024-03-31T15:56:28.734424+0000","last_undegraded":"2024-03-31T15:56:28.734424+0000","last_fullsized":"2024-03-31T15:56:28.734424+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:25.079313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047774+0000","last_change":"2024-03-31T15:56:26.036191+0000","last_active":"2024-03-31T15:56:28.047774+0000","last_peered":"2024-03-31T15:56:28.047774+0000","last_clean":"2024-03-31T15:56:28.047774+0000","last_became_active":"2024-03-31T15:56:26.035873+0000","last_became_peered":"2024-03-31T15:56:26.035873+0000","last_unstale":"2024-03-31T15:56:28.047774+0000","last_undegraded":"2024-03-31T15:56:28.047774+0000","last_fullsized":"2024-03-31T15:56:28.047774+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:31:04.170546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048941+0000","last_change":"2024-03-31T15:56:27.034376+0000","last_active":"2024-03-31T15:56:28.048941+0000","last_peered":"2024-03-31T15:56:28.048941+0000","last_clean":"2024-03-31T15:56:28.048941+0000","last_became_active":"2024-03-31T15:56:27.032545+0000","last_became_peered":"2024-03-31T15:56:27.032545+0000","last_unstale":"2024-03-31T15:56:28.048941+0000","last_undegraded":"2024-03-31T15:56:28.048941+0000","last_fullsized":"2024-03-31T15:56:28.048941+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731792+0000","last_change":"2024-03-31T15:56:26.044352+0000","last_active":"2024-03-31T15:56:28.731792+0000","last_peered":"2024-03-31T15:56:28.731792+0000","last_clean":"2024-03-31T15:56:28.731792+0000","last_became_active":"2024-03-31T15:56:26.044071+0000","last_became_peered":"2024-03-31T15:56:26.044071+0000","last_unstale":"2024-03-31T15:56:28.731792+0000","last_undegraded":"2024-03-31T15:56:28.731792+0000","last_fullsized":"2024-03-31T15:56:28.731792+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:09:52.264596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048061+0000","last_change":"2024-03-31T15:56:27.033778+0000","last_active":"2024-03-31T15:56:28.048061+0000","last_peered":"2024-03-31T15:56:28.048061+0000","last_clean":"2024-03-31T15:56:28.048061+0000","last_became_active":"2024-03-31T15:56:27.033343+0000","last_became_peered":"2024-03-31T15:56:27.033343+0000","last_unstale":"2024-03-31T15:56:28.048061+0000","last_undegraded":"2024-03-31T15:56:28.048061+0000","last_fullsized":"2024-03-31T15:56:28.048061+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:37:20.745451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734169+0000","last_change":"2024-03-31T15:56:26.030253+0000","last_active":"2024-03-31T15:56:28.734169+0000","last_peered":"2024-03-31T15:56:28.734169+0000","last_clean":"2024-03-31T15:56:28.734169+0000","last_became_active":"2024-03-31T15:56:26.029995+0000","last_became_peered":"2024-03-31T15:56:26.029995+0000","last_unstale":"2024-03-31T15:56:28.734169+0000","last_undegraded":"2024-03-31T15:56:28.734169+0000","last_fullsized":"2024-03-31T15:56:28.734169+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:45:21.317263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731894+0000","last_change":"2024-03-31T15:56:27.046229+0000","last_active":"2024-03-31T15:56:28.731894+0000","last_peered":"2024-03-31T15:56:28.731894+0000","last_clean":"2024-03-31T15:56:28.731894+0000","last_became_active":"2024-03-31T15:56:27.045950+0000","last_became_peered":"2024-03-31T15:56:27.045950+0000","last_unstale":"2024-03-31T15:56:28.731894+0000","last_undegraded":"2024-03-31T15:56:28.731894+0000","last_fullsized":"2024-03-31T15:56:28.731894+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T19:41:35.296800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731354+0000","last_change":"2024-03-31T15:56:26.037979+0000","last_active":"2024-03-31T15:56:28.731354+0000","last_peered":"2024-03-31T15:56:28.731354+0000","last_clean":"2024-03-31T15:56:28.731354+0000","last_became_active":"2024-03-31T15:56:26.037707+0000","last_became_peered":"2024-03-31T15:56:26.037707+0000","last_unstale":"2024-03-31T15:56:28.731354+0000","last_undegraded":"2024-03-31T15:56:28.731354+0000","last_fullsized":"2024-03-31T15:56:28.731354+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:25:28.079103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731185+0000","last_change":"2024-03-31T15:56:27.047264+0000","last_active":"2024-03-31T15:56:28.731185+0000","last_peered":"2024-03-31T15:56:28.731185+0000","last_clean":"2024-03-31T15:56:28.731185+0000","last_became_active":"2024-03-31T15:56:27.041690+0000","last_became_peered":"2024-03-31T15:56:27.041690+0000","last_unstale":"2024-03-31T15:56:28.731185+0000","last_undegraded":"2024-03-31T15:56:28.731185+0000","last_fullsized":"2024-03-31T15:56:28.731185+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:10:28.264724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.046922+0000","last_change":"2024-03-31T15:56:26.030797+0000","last_active":"2024-03-31T15:56:28.046922+0000","last_peered":"2024-03-31T15:56:28.046922+0000","last_clean":"2024-03-31T15:56:28.046922+0000","last_became_active":"2024-03-31T15:56:26.030470+0000","last_became_peered":"2024-03-31T15:56:26.030470+0000","last_unstale":"2024-03-31T15:56:28.046922+0000","last_undegraded":"2024-03-31T15:56:28.046922+0000","last_fullsized":"2024-03-31T15:56:28.046922+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:13:50.290565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048138+0000","last_change":"2024-03-31T15:56:27.052383+0000","last_active":"2024-03-31T15:56:28.048138+0000","last_peered":"2024-03-31T15:56:28.048138+0000","last_clean":"2024-03-31T15:56:28.048138+0000","last_became_active":"2024-03-31T15:56:27.052098+0000","last_became_peered":"2024-03-31T15:56:27.052098+0000","last_unstale":"2024-03-31T15:56:28.048138+0000","last_undegraded":"2024-03-31T15:56:28.048138+0000","last_fullsized":"2024-03-31T15:56:28.048138+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:30:59.047999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732818+0000","last_change":"2024-03-31T15:56:26.036552+0000","last_active":"2024-03-31T15:56:28.732818+0000","last_peered":"2024-03-31T15:56:28.732818+0000","last_clean":"2024-03-31T15:56:28.732818+0000","last_became_active":"2024-03-31T15:56:26.036319+0000","last_became_peered":"2024-03-31T15:56:26.036319+0000","last_unstale":"2024-03-31T15:56:28.732818+0000","last_undegraded":"2024-03-31T15:56:28.732818+0000","last_fullsized":"2024-03-31T15:56:28.732818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:09.232374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047730+0000","last_change":"2024-03-31T15:56:27.028944+0000","last_active":"2024-03-31T15:56:28.047730+0000","last_peered":"2024-03-31T15:56:28.047730+0000","last_clean":"2024-03-31T15:56:28.047730+0000","last_became_active":"2024-03-31T15:56:27.028674+0000","last_became_peered":"2024-03-31T15:56:27.028674+0000","last_unstale":"2024-03-31T15:56:28.047730+0000","last_undegraded":"2024-03-31T15:56:28.047730+0000","last_fullsized":"2024-03-31T15:56:28.047730+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T17:39:14.822208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"55'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732253+0000","last_change":"2024-03-31T15:56:26.039110+0000","last_active":"2024-03-31T15:56:28.732253+0000","last_peered":"2024-03-31T15:56:28.732253+0000","last_clean":"2024-03-31T15:56:28.732253+0000","last_became_active":"2024-03-31T15:56:26.037645+0000","last_became_peered":"2024-03-31T15:56:26.037645+0000","last_unstale":"2024-03-31T15:56:28.732253+0000","last_undegraded":"2024-03-31T15:56:28.732253+0000","last_fullsized":"2024-03-31T15:56:28.732253+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T00:51:27.467656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731739+0000","last_change":"2024-03-31T15:56:27.035617+0000","last_active":"2024-03-31T15:56:28.731739+0000","last_peered":"2024-03-31T15:56:28.731739+0000","last_clean":"2024-03-31T15:56:28.731739+0000","last_became_active":"2024-03-31T15:56:27.035325+0000","last_became_peered":"2024-03-31T15:56:27.035325+0000","last_unstale":"2024-03-31T15:56:28.731739+0000","last_undegraded":"2024-03-31T15:56:28.731739+0000","last_fullsized":"2024-03-31T15:56:28.731739+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:26:24.475678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231414+0000","last_change":"2024-03-31T15:56:26.040713+0000","last_active":"2024-03-31T15:56:28.231414+0000","last_peered":"2024-03-31T15:56:28.231414+0000","last_clean":"2024-03-31T15:56:28.231414+0000","last_became_active":"2024-03-31T15:56:26.040391+0000","last_became_peered":"2024-03-31T15:56:26.040391+0000","last_unstale":"2024-03-31T15:56:28.231414+0000","last_undegraded":"2024-03-31T15:56:28.231414+0000","last_fullsized":"2024-03-31T15:56:28.231414+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:48:09.232374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.732053+0000","last_change":"2024-03-31T15:56:27.055397+0000","last_active":"2024-03-31T15:56:28.732053+0000","last_peered":"2024-03-31T15:56:28.732053+0000","last_clean":"2024-03-31T15:56:28.732053+0000","last_became_active":"2024-03-31T15:56:27.054728+0000","last_became_peered":"2024-03-31T15:56:27.054728+0000","last_unstale":"2024-03-31T15:56:28.732053+0000","last_undegraded":"2024-03-31T15:56:28.732053+0000","last_fullsized":"2024-03-31T15:56:28.732053+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:30:31.753902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047898+0000","last_change":"2024-03-31T15:56:26.035712+0000","last_active":"2024-03-31T15:56:28.047898+0000","last_peered":"2024-03-31T15:56:28.047898+0000","last_clean":"2024-03-31T15:56:28.047898+0000","last_became_active":"2024-03-31T15:56:26.034040+0000","last_became_peered":"2024-03-31T15:56:26.034040+0000","last_unstale":"2024-03-31T15:56:28.047898+0000","last_undegraded":"2024-03-31T15:56:28.047898+0000","last_fullsized":"2024-03-31T15:56:28.047898+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:21:41.238108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.039713+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.037623+0000","last_became_peered":"2024-03-31T15:56:27.037623+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:23:10.759953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731486+0000","last_change":"2024-03-31T15:56:26.036984+0000","last_active":"2024-03-31T15:56:28.731486+0000","last_peered":"2024-03-31T15:56:28.731486+0000","last_clean":"2024-03-31T15:56:28.731486+0000","last_became_active":"2024-03-31T15:56:26.036674+0000","last_became_peered":"2024-03-31T15:56:26.036674+0000","last_unstale":"2024-03-31T15:56:28.731486+0000","last_undegraded":"2024-03-31T15:56:28.731486+0000","last_fullsized":"2024-03-31T15:56:28.731486+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:52:50.853047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.048608+0000","last_change":"2024-03-31T15:56:27.037261+0000","last_active":"2024-03-31T15:56:28.048608+0000","last_peered":"2024-03-31T15:56:28.048608+0000","last_clean":"2024-03-31T15:56:28.048608+0000","last_became_active":"2024-03-31T15:56:27.036282+0000","last_became_peered":"2024-03-31T15:56:27.036282+0000","last_unstale":"2024-03-31T15:56:28.048608+0000","last_undegraded":"2024-03-31T15:56:28.048608+0000","last_fullsized":"2024-03-31T15:56:28.048608+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T23:18:25.749230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.045745+0000","last_change":"2024-03-31T15:56:26.042322+0000","last_active":"2024-03-31T15:56:28.045745+0000","last_peered":"2024-03-31T15:56:28.045745+0000","last_clean":"2024-03-31T15:56:28.045745+0000","last_became_active":"2024-03-31T15:56:26.041942+0000","last_became_peered":"2024-03-31T15:56:26.041942+0000","last_unstale":"2024-03-31T15:56:28.045745+0000","last_undegraded":"2024-03-31T15:56:28.045745+0000","last_fullsized":"2024-03-31T15:56:28.045745+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:52:50.853047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731006+0000","last_change":"2024-03-31T15:56:27.034311+0000","last_active":"2024-03-31T15:56:28.731006+0000","last_peered":"2024-03-31T15:56:28.731006+0000","last_clean":"2024-03-31T15:56:28.731006+0000","last_became_active":"2024-03-31T15:56:27.033973+0000","last_became_peered":"2024-03-31T15:56:27.033973+0000","last_unstale":"2024-03-31T15:56:28.731006+0000","last_undegraded":"2024-03-31T15:56:28.731006+0000","last_fullsized":"2024-03-31T15:56:28.731006+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T21:15:39.134285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.731098+0000","last_change":"2024-03-31T15:56:26.041462+0000","last_active":"2024-03-31T15:56:28.731098+0000","last_peered":"2024-03-31T15:56:28.731098+0000","last_clean":"2024-03-31T15:56:28.731098+0000","last_became_active":"2024-03-31T15:56:26.041184+0000","last_became_peered":"2024-03-31T15:56:26.041184+0000","last_unstale":"2024-03-31T15:56:28.731098+0000","last_undegraded":"2024-03-31T15:56:28.731098+0000","last_fullsized":"2024-03-31T15:56:28.731098+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T16:25:11.564066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.734007+0000","last_change":"2024-03-31T15:56:27.042943+0000","last_active":"2024-03-31T15:56:28.734007+0000","last_peered":"2024-03-31T15:56:28.734007+0000","last_clean":"2024-03-31T15:56:28.734007+0000","last_became_active":"2024-03-31T15:56:27.042359+0000","last_became_peered":"2024-03-31T15:56:27.042359+0000","last_unstale":"2024-03-31T15:56:28.734007+0000","last_undegraded":"2024-03-31T15:56:28.734007+0000","last_fullsized":"2024-03-31T15:56:28.734007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:49:00.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733822+0000","last_change":"2024-03-31T15:56:26.036968+0000","last_active":"2024-03-31T15:56:28.733822+0000","last_peered":"2024-03-31T15:56:28.733822+0000","last_clean":"2024-03-31T15:56:28.733822+0000","last_became_active":"2024-03-31T15:56:26.036405+0000","last_became_peered":"2024-03-31T15:56:26.036405+0000","last_unstale":"2024-03-31T15:56:28.733822+0000","last_undegraded":"2024-03-31T15:56:28.733822+0000","last_fullsized":"2024-03-31T15:56:28.733822+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T20:28:07.298713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.733795+0000","last_change":"2024-03-31T15:56:27.057654+0000","last_active":"2024-03-31T15:56:28.733795+0000","last_peered":"2024-03-31T15:56:28.733795+0000","last_clean":"2024-03-31T15:56:28.733795+0000","last_became_active":"2024-03-31T15:56:27.057124+0000","last_became_peered":"2024-03-31T15:56:27.057124+0000","last_unstale":"2024-03-31T15:56:28.733795+0000","last_undegraded":"2024-03-31T15:56:28.733795+0000","last_fullsized":"2024-03-31T15:56:28.733795+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:26.009477+0000","last_clean_scrub_stamp":"2024-03-31T15:56:26.009477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:58:26.734092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.231103+0000","last_change":"2024-03-31T15:56:26.039648+0000","last_active":"2024-03-31T15:56:28.231103+0000","last_peered":"2024-03-31T15:56:28.231103+0000","last_clean":"2024-03-31T15:56:28.231103+0000","last_became_active":"2024-03-31T15:56:26.039276+0000","last_became_peered":"2024-03-31T15:56:26.039276+0000","last_unstale":"2024-03-31T15:56:28.231103+0000","last_undegraded":"2024-03-31T15:56:28.231103+0000","last_fullsized":"2024-03-31T15:56:28.231103+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:56:25.006583+0000","last_clean_scrub_stamp":"2024-03-31T15:56:25.006583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:52:50.853047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-31T15:56:28.047349+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:28.047349+0000","last_peered":"2024-03-31T15:56:28.047349+0000","last_clean":"2024-03-31T15:56:28.047349+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:28.047349+0000","last_undegraded":"2024-03-31T15:56:28.047349+0000","last_fullsized":"2024-03-31T15:56:28.047349+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509478,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":294340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509478,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":659456,"data_stored":294992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509479,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":663552,"data_stored":295952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509479,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":655360,"data_stored":294430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509479,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1261568,"data_stored":886902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542182,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542182,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":659456,"data_stored":294396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1253376,"data_stored":884868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:56:36.121 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-31T15:56:36.121 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-31T15:56:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.308+0000 7fb387e00640 1 Processor -- start 2024-03-31T15:56:36.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.308+0000 7fb387e00640 1 -- start start 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb38815fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb38815d2d0 con 0x7fb3880d16f0 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3880cb6b0 con 0x7fb38815bee0 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb388070030 con 0x7fb3880cb220 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb386000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb3880d16f0 0x7fb38806b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38174/0 (socket says 172.21.15.90:38174) 2024-03-31T15:56:36.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb386000640 1 -- 172.21.15.90:0/2071482973 learned_addr learned my addr 172.21.15.90:0/2071482973 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb38815fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1773586644 0 0) 0x7fb38815d2d0 con 0x7fb3880d16f0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb36c003530 con 0x7fb3880d16f0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012486434 0 0) 0x7fb378001120 con 0x7fb38815bee0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb38815d2d0 con 0x7fb38815bee0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1037855753 0 0) 0x7fb36c003530 con 0x7fb3880d16f0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 msgr2=0x7fb38815fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb38815fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 >> v1:172.21.15.102:6789/0 conn(0x7fb38815bee0 legacy=0x7fb388072770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38806fa70 con 0x7fb3880d16f0 2024-03-31T15:56:36.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fb374001680 con 0x7fb3880d16f0 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 >> v1:172.21.15.90:6789/0 conn(0x7fb3880d16f0 legacy=0x7fb38806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb36c003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb36c004130 0x7fb36c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb36c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb388070030 con 0x7fb36c004d50 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c00bc30 con 0x7fb36c004130 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c00be30 con 0x7fb3880cb220 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb36c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb386000640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb36c004130 0x7fb36c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb386a00640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb36c003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 -- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 msgr2=0x7fb36c003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb36c003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 -- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb36c004130 msgr2=0x7fb36c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:36.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb36c004130 0x7fb36c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb387400640 1 -- 172.21.15.90:0/2071482973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c003530 con 0x7fb36c004d50 2024-03-31T15:56:36.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.312+0000 7fb386a00640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880cb220 0x7fb36c003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387400640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb36c0051d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb380009fc0 tx=0x7fb38001cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d555fa8caa29899 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3800b2070 con 0x7fb36c004d50 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3800a5c40 con 0x7fb36c004d50 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb385600640 1 -- 172.21.15.90:0/2071482973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380009c10 con 0x7fb36c004d50 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- 172.21.15.90:0/2071482973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 msgr2=0x7fb36c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 --2- 172.21.15.90:0/2071482973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb36c0051d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb380009fc0 tx=0x7fb38001cc50 comp rx=0 tx=0).stop 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387400640 1 -- 172.21.15.90:0/2071482973 reap_dead start 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- 172.21.15.90:0/2071482973 shutdown_connections 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- 172.21.15.90:0/2071482973 >> 172.21.15.90:0/2071482973 conn(0x7fb3880d0bd0 msgr2=0x7fb3880d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- 172.21.15.90:0/2071482973 shutdown_connections 2024-03-31T15:56:36.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- 172.21.15.90:0/2071482973 wait complete. 2024-03-31T15:56:36.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 Processor -- start 2024-03-31T15:56:36.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.316+0000 7fb387e00640 1 -- start start 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880c6120 0x7fb3880c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 0x7fb38806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb388073000 con 0x7fb36c004d50 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb388072e80 con 0x7fb38806e9b0 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fb38806bd00 con 0x7fb3880c6120 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59620/0 (socket says 172.21.15.90:59620) 2024-03-31T15:56:36.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 -- 172.21.15.90:0/4082603683 learned_addr learned my addr 172.21.15.90:0/4082603683 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 -- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880c6120 msgr2=0x7fb3880c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386000640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880c6120 0x7fb3880c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880c6120 0x7fb3880c65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 -- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 msgr2=0x7fb38806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387400640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 0x7fb38806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 0x7fb38806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 -- 172.21.15.90:0/4082603683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb388157240 con 0x7fb36c004d50 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387400640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 0x7fb38806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb386a00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb37800e740 tx=0x7fb37800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3780050d0 con 0x7fb36c004d50 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb378005270 con 0x7fb36c004d50 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3881574e0 con 0x7fb36c004d50 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378005460 con 0x7fb36c004d50 2024-03-31T15:56:36.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb388157aa0 con 0x7fb36c004d50 2024-03-31T15:56:36.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.320+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb388068d00 con 0x7fb36c004d50 2024-03-31T15:56:36.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.324+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb37801a020 con 0x7fb36c004d50 2024-03-31T15:56:36.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.324+0000 7fb37ea00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 0x7fb36806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.324+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fb3780862e0 con 0x7fb36c004d50 2024-03-31T15:56:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.324+0000 7fb386000640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 0x7fb36806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.324+0000 7fb386000640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 0x7fb36806a840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb388157080 tx=0x7fb374009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.328+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fb3880cb6b0 con 0x7fb36c004d50 2024-03-31T15:56:36.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.540+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb388073200 con 0x7fb36c004d50 2024-03-31T15:56:36.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.544+0000 7fb37ea00640 1 -- 172.21.15.90:0/4082603683 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7fb3780567a0 con 0x7fb36c004d50 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-31T15:56:36.538 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.544+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 msgr2=0x7fb36806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.544+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 0x7fb36806a840 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb388157080 tx=0x7fb374009040 comp rx=0 tx=0).stop 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 msgr2=0x7fb3881527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb37800e740 tx=0x7fb37800ec00 comp rx=0 tx=0).stop 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 shutdown_connections 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fb368068380 0x7fb36806a840 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb38806e9b0 0x7fb38806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fb3880c6120 0x7fb3880c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 --2- 172.21.15.90:0/4082603683 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb36c004d50 0x7fb3881527e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 >> 172.21.15.90:0/4082603683 conn(0x7fb3880d0bd0 msgr2=0x7fb38815dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 shutdown_connections 2024-03-31T15:56:36.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.548+0000 7fb387e00640 1 -- 172.21.15.90:0/4082603683 wait complete. 2024-03-31T15:56:36.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:36.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 Processor -- start 2024-03-31T15:56:36.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 -- start start 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fae641449a0 0x7fae6415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae6415ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae640b47e0 con 0x7fae640b3d90 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fae640b4ac0 con 0x7fae641449a0 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae69a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fae6415b7d0 con 0x7fae6406bf30 2024-03-31T15:56:36.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fae641449a0 0x7fae6415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fae640b3d90 0x7fae6406b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38184/0 (socket says 172.21.15.90:38184) 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62000640 1 -- 172.21.15.90:0/1440774459 learned_addr learned my addr 172.21.15.90:0/1440774459 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae63400640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae6415ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae61600640 1 -- 172.21.15.90:0/1440774459 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1969394995 0 0) 0x7fae640b47e0 con 0x7fae640b3d90 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae61600640 1 -- 172.21.15.90:0/1440774459 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae48005740 con 0x7fae640b3d90 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62a00640 1 -- 172.21.15.90:0/1440774459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 msgr2=0x7fae6415ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62a00640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae6415ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62a00640 1 -- 172.21.15.90:0/1440774459 >> v1:172.21.15.90:6789/0 conn(0x7fae640b3d90 legacy=0x7fae6406b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.712+0000 7fae62a00640 1 -- 172.21.15.90:0/1440774459 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6415bab0 con 0x7fae641449a0 2024-03-31T15:56:36.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae62a00640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae641449a0 0x7fae6415bfd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fae540097c0 tx=0x7fae5401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6f0d7575febcf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae61600640 1 -- 172.21.15.90:0/1440774459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae540b1070 con 0x7fae641449a0 2024-03-31T15:56:36.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae61600640 1 -- 172.21.15.90:0/1440774459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae540ac410 con 0x7fae641449a0 2024-03-31T15:56:36.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae61600640 1 -- 172.21.15.90:0/1440774459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae540ac7e0 con 0x7fae641449a0 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 -- 172.21.15.90:0/1440774459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae641449a0 msgr2=0x7fae6415bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae641449a0 0x7fae6415bfd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fae540097c0 tx=0x7fae5401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 -- 172.21.15.90:0/1440774459 shutdown_connections 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae6415ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 --2- 172.21.15.90:0/1440774459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae641449a0 0x7fae6415bfd0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 -- 172.21.15.90:0/1440774459 >> 172.21.15.90:0/1440774459 conn(0x7fae640b1c80 msgr2=0x7fae640b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:36.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 -- 172.21.15.90:0/1440774459 shutdown_connections 2024-03-31T15:56:36.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.720+0000 7fae69a00640 1 -- 172.21.15.90:0/1440774459 wait complete. 2024-03-31T15:56:36.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 Processor -- start 2024-03-31T15:56:36.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- start start 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae641433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 0x7fae641591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fae640b3010 con 0x7fae640b3d90 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae640b2e90 con 0x7fae641449a0 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fae640b3190 con 0x7fae6406bf30 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59640/0 (socket says 172.21.15.90:59640) 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 -- 172.21.15.90:0/3799838660 learned_addr learned my addr 172.21.15.90:0/3799838660 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae63400640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 0x7fae641591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae641433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 -- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 msgr2=0x7fae641433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae641433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 -- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 msgr2=0x7fae641591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 0x7fae641591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 -- 172.21.15.90:0/3799838660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae64143930 con 0x7fae640b3d90 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae63400640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 0x7fae641591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae62000640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c450 tx=0x7fae5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50017070 con 0x7fae640b3d90 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae50005cf0 con 0x7fae640b3d90 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae500072f0 con 0x7fae640b3d90 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae64164530 con 0x7fae640b3d90 2024-03-31T15:56:36.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.724+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae641fac50 con 0x7fae640b3d90 2024-03-31T15:56:36.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae640686d0 con 0x7fae640b3d90 2024-03-31T15:56:36.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fae5001a070 con 0x7fae640b3d90 2024-03-31T15:56:36.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae5aa00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 0x7fae4406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:36.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fae50085f20 con 0x7fae640b3d90 2024-03-31T15:56:36.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae62a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 0x7fae4406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:36.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.728+0000 7fae62a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 0x7fae4406a840 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fae540ae000 tx=0x7fae54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:36.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.736+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fae50055cf0 con 0x7fae640b3d90 2024-03-31T15:56:36.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae64072050 con 0x7fae640b3d90 2024-03-31T15:56:36.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae5aa00640 1 -- 172.21.15.90:0/3799838660 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7fae50059ff0 con 0x7fae640b3d90 2024-03-31T15:56:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:36.921 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:36.922 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:36.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 msgr2=0x7fae4406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 0x7fae4406a840 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fae540ae000 tx=0x7fae54008040 comp rx=0 tx=0).stop 2024-03-31T15:56:36.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 msgr2=0x7fae6413c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:36.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fae5000c450 tx=0x7fae5000c910 comp rx=0 tx=0).stop 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 shutdown_connections 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fae44068380 0x7fae4406a840 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae641449a0 0x7fae641591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae640b3d90 0x7fae6413c9c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 --2- 172.21.15.90:0/3799838660 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fae6406bf30 0x7fae641433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 >> 172.21.15.90:0/3799838660 conn(0x7fae640b1c80 msgr2=0x7fae6415e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 shutdown_connections 2024-03-31T15:56:36.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:36.928+0000 7fae69a00640 1 -- 172.21.15.90:0/3799838660 wait complete. 2024-03-31T15:56:37.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-31T15:56:26.396713+0000', 'modified': '2024-03-31T15:56:29.426781+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6213}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6213': {'gid': 6213, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6835/1583024065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6833', 'nonce': 1583024065}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 1583024065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6213, 'qdb_cluster': [6213]} 2024-03-31T15:56:37.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-31T15:56:37.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-31T15:56:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 Processor -- start 2024-03-31T15:56:37.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 -- start start 2024-03-31T15:56:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f65d40d0d20 0x7f65d40d2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d40d1190 con 0x7f65d415e320 2024-03-31T15:56:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d40d1400 con 0x7f65d41624a0 2024-03-31T15:56:37.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d8e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f65d415e700 con 0x7f65d40d0d20 2024-03-31T15:56:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f65d415e320 0x7f65d4161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38186/0 (socket says 172.21.15.90:38186) 2024-03-31T15:56:37.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.492+0000 7f65d1e00640 1 -- 172.21.15.90:0/3415679156 learned_addr learned my addr 172.21.15.90:0/3415679156 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> v2:172.21.15.102:3300/0 conn(0x7f65d40d0d20 0x7f65d40d2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953698721 0 0) 0x7f65d40d1190 con 0x7f65d415e320 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65b8003530 con 0x7f65d415e320 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3948396866 0 0) 0x7f65d40d1400 con 0x7f65d41624a0 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65d40d1190 con 0x7f65d41624a0 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3592663933 0 0) 0x7f65b8003530 con 0x7f65d415e320 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 >> v2:172.21.15.102:3300/0 conn(0x7f65d40d0d20 msgr2=0x7f65d40d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 --2- 172.21.15.90:0/3415679156 >> v2:172.21.15.102:3300/0 conn(0x7f65d40d0d20 0x7f65d40d2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 >> v1:172.21.15.102:6789/0 conn(0x7f65d41624a0 legacy=0x7f65d40b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d40b1a20 con 0x7f65d415e320 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f65c4001680 con 0x7f65d415e320 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> v2:172.21.15.102:3300/0 conn(0x7f65d40d0d20 0x7f65d40d2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 >> v1:172.21.15.90:6789/0 conn(0x7f65d415e320 legacy=0x7f65d4161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65b8003b50 0x7f65b8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b8004810 0x7f65b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 0x7f65b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65b800c020 con 0x7f65b80051d0 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65b800c1d0 con 0x7f65b8004810 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f65b800c3d0 con 0x7f65b8003b50 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 0x7f65b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65b8003b50 0x7f65b8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b8004810 0x7f65b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 -- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65b8003b50 msgr2=0x7f65b8003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65b8003b50 0x7f65b8003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 -- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b8004810 msgr2=0x7f65b8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b8004810 0x7f65b8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.496+0000 7f65d2800640 1 -- 172.21.15.90:0/3415679156 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b8003530 con 0x7f65b80051d0 2024-03-31T15:56:37.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d2800640 1 -- 172.21.15.90:0/3415679156 reap_dead start 2024-03-31T15:56:37.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d2800640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 0x7f65b800b9d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e350 tx=0x7f65cc01ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14e41c02c47fabe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc009de0 con 0x7f65b80051d0 2024-03-31T15:56:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc0039a0 con 0x7f65b80051d0 2024-03-31T15:56:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d1400640 1 -- 172.21.15.90:0/3415679156 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc003d90 con 0x7f65b80051d0 2024-03-31T15:56:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 msgr2=0x7f65b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 0x7f65b800b9d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e350 tx=0x7f65cc01ce80 comp rx=0 tx=0).stop 2024-03-31T15:56:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- 172.21.15.90:0/3415679156 shutdown_connections 2024-03-31T15:56:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3415679156 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65b80051d0 0x7f65b800b9d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- 172.21.15.90:0/3415679156 >> 172.21.15.90:0/3415679156 conn(0x7f65d408bfd0 msgr2=0x7f65d40d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- 172.21.15.90:0/3415679156 shutdown_connections 2024-03-31T15:56:37.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- 172.21.15.90:0/3415679156 wait complete. 2024-03-31T15:56:37.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 Processor -- start 2024-03-31T15:56:37.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.500+0000 7f65d8e00640 1 -- start start 2024-03-31T15:56:37.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 0x7f65d4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 0x7f65d41f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65d40d2940 con 0x7f65d41602a0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65d40b1fe0 con 0x7f65b80051d0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f65d40d2ac0 con 0x7f65d41f3ca0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59658/0 (socket says 172.21.15.90:59658) 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 -- 172.21.15.90:0/3448353330 learned_addr learned my addr 172.21.15.90:0/3448353330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 0x7f65d4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d2800640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 0x7f65d41f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 -- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 msgr2=0x7f65d41f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 0x7f65d41f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 -- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 msgr2=0x7f65d4149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 0x7f65d4149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 -- 172.21.15.90:0/3448353330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d41f4a10 con 0x7f65d41602a0 2024-03-31T15:56:37.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 0x7f65d4149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:37.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d2800640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 0x7f65d41f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:37.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00f770 tx=0x7f65bc00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65ca000640 1 -- 172.21.15.90:0/3448353330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc01a070 con 0x7f65d41602a0 2024-03-31T15:56:37.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65ca000640 1 -- 172.21.15.90:0/3448353330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc005110 con 0x7f65d41602a0 2024-03-31T15:56:37.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65ca000640 1 -- 172.21.15.90:0/3448353330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc007030 con 0x7f65d41602a0 2024-03-31T15:56:37.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d41f4d10 con 0x7f65d41602a0 2024-03-31T15:56:37.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.504+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d41f52d0 con 0x7f65d41602a0 2024-03-31T15:56:37.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65ca000640 1 -- 172.21.15.90:0/3448353330 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f65bc0073e0 con 0x7f65d41602a0 2024-03-31T15:56:37.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65ca000640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 0x7f65b006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65ca000640 1 -- 172.21.15.90:0/3448353330 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f65bc086550 con 0x7f65d41602a0 2024-03-31T15:56:37.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 Processor -- start 2024-03-31T15:56:37.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 0x7f65b006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 -- start start 2024-03-31T15:56:37.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 0x7f65b006a870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f65c4001780 tx=0x7f65c4001e60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 0x560e4359a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560e433a0950 con 0x560e4354e0b0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560e4333d990 con 0x560e4359adc0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x560e4339c900 con 0x560e4359a4a0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59662/0 (socket says 172.21.15.90:59662) 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 -- 172.21.15.90:0/3094639598 learned_addr learned my addr 172.21.15.90:0/3094639598 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 0x560e4359a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 -- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 msgr2=0x560e4359a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 0x560e4359a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 -- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 msgr2=0x560e4354db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 -- 172.21.15.90:0/3094639598 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e433e5100 con 0x560e4354e0b0 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65cbe00640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d1e00640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 0x560e4359a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.508+0000 7f65d2800640 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f65cc01d360 tx=0x7f65cc01b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1bc06e7180fca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65c2a00640 1 -- 172.21.15.90:0/3094639598 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc003ba0 con 0x560e4354e0b0 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65c2a00640 1 -- 172.21.15.90:0/3094639598 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc003d40 con 0x560e4354e0b0 2024-03-31T15:56:37.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65c2a00640 1 -- 172.21.15.90:0/3094639598 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc004220 con 0x560e4354e0b0 2024-03-31T15:56:37.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/3094639598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 msgr2=0x560e4354e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f65cc01d360 tx=0x7f65cc01b6c0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/3094639598 shutdown_connections 2024-03-31T15:56:37.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4354e0b0 0x560e4354e490 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- 172.21.15.90:0/3094639598 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4359a4a0 0x560e4359a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/3094639598 >> 172.21.15.90:0/3094639598 conn(0x560e43438b80 msgr2=0x560e4357d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/3094639598 shutdown_connections 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/3094639598 wait complete. 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- start start 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 Processor -- start 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4355ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4355b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560e4339fc60 con 0x560e4359a4a0 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560e4335e3e0 con 0x560e4359adc0 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x560e433ad060 con 0x560e4354e0b0 2024-03-31T15:56:37.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 10 client.0 _ll_register_callbacks cb 0x560e434010f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59676/0 (socket says 172.21.15.90:59676) 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 -- 172.21.15.90:0/408490627 learned_addr learned my addr 172.21.15.90:0/408490627 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65d1e00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4355ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65d2800640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4355b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 -- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 msgr2=0x560e4355ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4355ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 -- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 msgr2=0x560e4355b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4355b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 -- 172.21.15.90:0/408490627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e4336a8c0 con 0x560e4359a4a0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65cbe00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007d10 tx=0x7f65bc0569a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba147be44580638 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65b7400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:56:37.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0862b0 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f65bc05a0b0 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.512+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65cc003d40 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f65bc05edc0 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 10 client.6372 fetch_fsmap learned FSMap version 27 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x560e43328e80 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc003d40 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7f65bc055630 con 0x560e4359a4a0 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 10 client.6372 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 10 client.6372 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 20 client.6372 populate_metadata read hostname 'smithi090' 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4355c120 0x560e435aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 4 client.6372 mds_command: new command op to 6213 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 --> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e4333bbc0 con 0x560e4355c120 2024-03-31T15:56:37.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65d1e00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4355c120 0x560e435aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65d1e00640 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4355c120 0x560e435aca20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48cf1b0386b1eacb server_cookie=b3f5b84542559a60 in_seq=0 out_seq=0 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 10 client.6372 ms_handle_connect on v2:172.21.15.90:6833/1583024065 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 1 -- 172.21.15.90:0/408490627 <== mds.0 v2:172.21.15.90:6833/1583024065 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e4333bbc0 con 0x560e4355c120 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 10 client.6372 handle_command_reply: tid=0 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 client.6372 shutdown 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 msgr2=0x560e4355b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007d10 tx=0x7f65bc0569a0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 shutdown_connections 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4355c120 0x560e435aca20 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4355b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4355b340 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65b7400640 0 client.6372 ms_handle_reset on v2:172.21.15.90:6833/1583024065 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 --2- 172.21.15.90:0/408490627 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4355ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 >> 172.21.15.90:0/408490627 conn(0x560e43438b80 msgr2=0x560e4354d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 shutdown_connections 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 1 -- 172.21.15.90:0/408490627 wait complete. 2024-03-31T15:56:37.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.516+0000 7f65dc302000 20 client.6372 trim_cache size 0 max 16384 2024-03-31T15:56:37.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 Processor -- start 2024-03-31T15:56:37.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- start start 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4356ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65d42010b0 con 0x560e4359a4a0 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65d42000d0 con 0x560e4359adc0 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f65d41ff430 con 0x560e4354e0b0 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59692/0 (socket says 172.21.15.90:59692) 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 -- 172.21.15.90:0/155436631 learned_addr learned my addr 172.21.15.90:0/155436631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65d1e00640 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4356ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65d2800640 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 -- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 msgr2=0x560e4356ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4356ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 -- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 msgr2=0x560e4354d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 -- 172.21.15.90:0/155436631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e4335b490 con 0x560e4359a4a0 2024-03-31T15:56:37.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65cbe00640 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007a10 tx=0x7f65bc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac004e55e0534b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65c2000640 1 -- 172.21.15.90:0/155436631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc0860b0 con 0x560e4359a4a0 2024-03-31T15:56:37.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65c2000640 1 -- 172.21.15.90:0/155436631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc086250 con 0x560e4359a4a0 2024-03-31T15:56:37.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65c2000640 1 -- 172.21.15.90:0/155436631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc05a2b0 con 0x560e4359a4a0 2024-03-31T15:56:37.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- 172.21.15.90:0/155436631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 msgr2=0x560e4356b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f65bc007a10 tx=0x7f65bc015000 comp rx=0 tx=0).stop 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- 172.21.15.90:0/155436631 shutdown_connections 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359adc0 0x560e4354d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359a4a0 0x560e4356b390 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 --2- 172.21.15.90:0/155436631 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e4356ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- 172.21.15.90:0/155436631 >> 172.21.15.90:0/155436631 conn(0x560e43438b80 msgr2=0x560e435a7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- 172.21.15.90:0/155436631 shutdown_connections 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.536+0000 7f65dc302000 1 -- 172.21.15.90:0/155436631 wait complete. 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- start start 2024-03-31T15:56:37.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 Processor -- start 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e436070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 0x560e4357bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560e433a4f80 con 0x560e4359adc0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65d41ffa80 con 0x560e4359a4a0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x560e43368300 con 0x560e4354e0b0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 10 client.0 _ll_register_callbacks cb 0x7f65bc0076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d1e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e436070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59700/0 (socket says 172.21.15.90:59700) 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 0x560e4357bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 -- 172.21.15.90:0/196735856 learned_addr learned my addr 172.21.15.90:0/196735856 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 -- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 msgr2=0x560e436070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e436070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 -- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 msgr2=0x560e4357bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 0x560e4357bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 -- 172.21.15.90:0/196735856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e43359e80 con 0x560e4359adc0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d1e00640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e436070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65cbe00640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 0x560e4357bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d2800640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0b8040 tx=0x7f65b8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05f9c3be8989f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0039a0 con 0x560e4359adc0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f65cc003b40 con 0x560e4359adc0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65bc086250 con 0x560e4359adc0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f65cc0d1050 con 0x560e4359adc0 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 10 client.6378 fetch_fsmap learned FSMap version 27 2024-03-31T15:56:37.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) -- 0x560e43347270 con 0x560e4359adc0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc086250 con 0x560e4359adc0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 27) ==== 3494+0+0 (secure 0 0 0) 0x7f65cc0d4020 con 0x560e4359adc0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 10 client.6378 fetch_fsmap finished waiting for FSMap version 27 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 10 client.6378 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 20 client.6378 populate_metadata read hostname 'smithi090' 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4357c620 0x560e4357ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 4 client.6378 mds_command: new command op to 6213 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 --> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] -- command(tid 0: {"prefix": "status"}) -- 0x560e433ade90 con 0x560e4357c620 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.540+0000 7f65d1e00640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4357c620 0x560e4357ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65b7400640 10 client.6378 ms_handle_connect on v2:172.21.15.90:6833/1583024065 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d1e00640 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4357c620 0x560e4357ca00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae1ff5acb007311 server_cookie=779f9ed2c26ba4f1 in_seq=0 out_seq=0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65b7400640 1 -- 172.21.15.90:0/196735856 <== mds.0 v2:172.21.15.90:6833/1583024065 1 ==== command_reply(tid 0: 0 ) ==== 8+0+317 (crc 0 0 0) 0x560e433ade90 con 0x560e4357c620 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65b7400640 10 client.6378 handle_command_reply: tid=0 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 client.6378 shutdown 2024-03-31T15:56:37.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 msgr2=0x560e4357c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0b8040 tx=0x7f65b8005840 comp rx=0 tx=0).stop 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 shutdown_connections 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:6833/1583024065,v1:172.21.15.90:6835/1583024065] conn(0x560e4357c620 0x560e4357ca00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x560e4359adc0 0x560e4357c0e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560e4359a4a0 0x560e4357bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 --2- 172.21.15.90:0/196735856 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x560e4354e0b0 0x560e436070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65b7400640 0 client.6378 ms_handle_reset on v2:172.21.15.90:6833/1583024065 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 >> 172.21.15.90:0/196735856 conn(0x560e43438b80 msgr2=0x560e4356ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 shutdown_connections 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 1 -- 172.21.15.90:0/196735856 wait complete. 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65dc302000 20 client.6378 trim_cache size 0 max 16384 2024-03-31T15:56:37.541 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "0135ca30-606a-448f-b61f-3ca22bf9710a", 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6213, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs", 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 10.527945568, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 27, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 55, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 55, 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 15.4797689 2024-03-31T15:56:37.542 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 msgr2=0x7f65b006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 0x7f65b006a870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f65c4001780 tx=0x7f65c4001e60 comp rx=0 tx=0).stop 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 msgr2=0x7f65d4160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00f770 tx=0x7f65bc00fc30 comp rx=0 tx=0).stop 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 shutdown_connections 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f65b00683b0 0x7f65b006a870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f65d41f3ca0 0x7f65d41f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65d41602a0 0x7f65d4160720 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 --2- 172.21.15.90:0/3448353330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80051d0 0x7f65d4149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 >> 172.21.15.90:0/3448353330 conn(0x7f65d408bfd0 msgr2=0x7f65d40d0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 shutdown_connections 2024-03-31T15:56:37.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.544+0000 7f65d8e00640 1 -- 172.21.15.90:0/3448353330 wait complete. 2024-03-31T15:56:37.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:56:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 Processor -- start 2024-03-31T15:56:37.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 -- start start 2024-03-31T15:56:37.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33700d2680 0x7f33700b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3370160350 con 0x7f33700b2c20 2024-03-31T15:56:37.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3370160bf0 con 0x7f33700d2680 2024-03-31T15:56:37.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3376e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3370160630 con 0x7f337015c120 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:56054/0 (socket says 172.21.15.90:56054) 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 -- 172.21.15.90:0/2490980185 learned_addr learned my addr 172.21.15.90:0/2490980185 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f3367e00640 1 --2- 172.21.15.90:0/2490980185 >> v2:172.21.15.102:3300/0 conn(0x7f33700d2680 0x7f33700b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 -- 172.21.15.90:0/2490980185 >> v2:172.21.15.102:3300/0 conn(0x7f33700d2680 msgr2=0x7f33700b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 --2- 172.21.15.90:0/2490980185 >> v2:172.21.15.102:3300/0 conn(0x7f33700d2680 0x7f33700b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 -- 172.21.15.90:0/2490980185 >> v1:172.21.15.102:6789/0 conn(0x7f33700b2c20 legacy=0x7f33700cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.700+0000 7f336fe00640 1 -- 172.21.15.90:0/2490980185 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3370160910 con 0x7f337015c120 2024-03-31T15:56:37.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f336fe00640 1 --2- 172.21.15.90:0/2490980185 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bec58fe43ddbc6d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f336f400640 1 -- 172.21.15.90:0/2490980185 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600ab430 con 0x7f337015c120 2024-03-31T15:56:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f336f400640 1 -- 172.21.15.90:0/2490980185 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33600b0070 con 0x7f337015c120 2024-03-31T15:56:37.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f336f400640 1 -- 172.21.15.90:0/2490980185 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600ab7f0 con 0x7f337015c120 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 -- 172.21.15.90:0/2490980185 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 msgr2=0x7f337015fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 --2- 172.21.15.90:0/2490980185 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336001c410 comp rx=0 tx=0).stop 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 -- 172.21.15.90:0/2490980185 shutdown_connections 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 --2- 172.21.15.90:0/2490980185 >> v2:172.21.15.102:3300/0 conn(0x7f33700d2680 0x7f33700b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 --2- 172.21.15.90:0/2490980185 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f337015fd00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 -- 172.21.15.90:0/2490980185 >> 172.21.15.90:0/2490980185 conn(0x7f337009aa40 msgr2=0x7f337009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 -- 172.21.15.90:0/2490980185 shutdown_connections 2024-03-31T15:56:37.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.704+0000 7f3376e00640 1 -- 172.21.15.90:0/2490980185 wait complete. 2024-03-31T15:56:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 Processor -- start 2024-03-31T15:56:37.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 -- start start 2024-03-31T15:56:37.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33700d2680 0x7f33701544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f33701587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f33701691c0 con 0x7f33700b2c20 2024-03-31T15:56:37.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3370169040 con 0x7f33700d2680 2024-03-31T15:56:37.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f3376e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3370169340 con 0x7f337015c120 2024-03-31T15:56:37.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59704/0 (socket says 172.21.15.90:59704) 2024-03-31T15:56:37.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 -- 172.21.15.90:0/3725318362 learned_addr learned my addr 172.21.15.90:0/3725318362 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 -- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 msgr2=0x7f33701587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f33701587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 -- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33700d2680 msgr2=0x7f33701544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33700d2680 0x7f33701544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.708+0000 7f336fe00640 1 -- 172.21.15.90:0/3725318362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33700cee90 con 0x7f33700b2c20 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336fe00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f336001aac0 tx=0x7f33600191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600abd70 con 0x7f33700b2c20 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33600be410 con 0x7f33700b2c20 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33700b0040 con 0x7f33700b2c20 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600be800 con 0x7f33700b2c20 2024-03-31T15:56:37.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33700b0600 con 0x7f33700b2c20 2024-03-31T15:56:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33700686d0 con 0x7f33700b2c20 2024-03-31T15:56:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f33600bc020 con 0x7f33700b2c20 2024-03-31T15:56:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 0x7f334c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f3360120ca0 con 0x7f33700b2c20 2024-03-31T15:56:37.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.712+0000 7f3367e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 0x7f334c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:37.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.716+0000 7f3367e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 0x7f334c06a840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3358006aa0 tx=0x7f3358006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:37.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.716+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3360016150 con 0x7f33700b2c20 2024-03-31T15:56:37.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.908+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f33700b1b60 con 0x7f33700b2c20 2024-03-31T15:56:37.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.908+0000 7f336cc00640 1 -- 172.21.15.90:0/3725318362 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f33600f1030 con 0x7f33700b2c20 2024-03-31T15:56:37.904 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 msgr2=0x7f334c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 0x7f334c06a840 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3358006aa0 tx=0x7f3358006a30 comp rx=0 tx=0).stop 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 msgr2=0x7f33700af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f336001aac0 tx=0x7f33600191b0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 shutdown_connections 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f334c068380 0x7f334c06a840 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f337015c120 0x7f33701587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33700d2680 0x7f33701544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 --2- 172.21.15.90:0/3725318362 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33700b2c20 0x7f33700af940 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:37.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 >> 172.21.15.90:0/3725318362 conn(0x7f337009aa40 msgr2=0x7f33700ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:37.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 shutdown_connections 2024-03-31T15:56:37.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:37.912+0000 7f3376e00640 1 -- 172.21.15.90:0/3725318362 wait complete. 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:56:37.931 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:56:37.932 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:37.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:37.932 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.941 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42325sec preferred_lft 42325sec 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.942 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.943 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:37.944 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:37.944 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:56:37.944 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:37.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:37.944 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:56:38.001 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:56:38.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.003 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:38.055 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:56:38.056 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:56:38.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:56:38.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:38.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.190 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:56:38.198 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:56:38.198 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:56:38.199 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:38.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:56:38.262 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:56:38.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:38.263 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:38.311 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:38.311 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:38.311 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:38.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:38.423 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:38.472 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:38.473 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:38.474 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:38.475 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:38.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.476 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:38.527 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:56:38.527 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:56:38.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:38.591 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:38.641 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:38.642 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:38.643 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:38.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.644 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:38.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:56:38.668+0000 7f61f53d0580 -1 init, newargv = 0x557fa50522b0 newargc=15 2024-03-31T15:56:38.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[41631]: starting ceph client 2024-03-31T15:56:38.669 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[41631]: starting fuse 2024-03-31T15:56:38.671 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:56:38.671 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:56:38.671 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:56:38.671 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:56:38.671 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:56:38.671 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:56:38.671 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:56:38.672 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:56:38.673 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:56:38.796 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.41631.asok 2024-03-31T15:56:38.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:38.807 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41631.asok 2024-03-31T15:56:38.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.807 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41631.asok status 2024-03-31T15:56:38.980 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:56:38.980 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:56:38.980 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:56:38.980 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:56:38.980 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "41631", 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6390, 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6390 2024-03-31T15:56:38.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270261191 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2270261191 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6390 192.168.0.1:0/2270261191", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/2270261191", 2024-03-31T15:56:38.982 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 27, 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 55, 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:56:38.983 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:56:38.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:38.995 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:56:38.995 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:39.006 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:56:39.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:39.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:39.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:39.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:39.073 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-31T15:56:39.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-31T15:56:39.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.324+0000 7fab7e200640 1 Processor -- start 2024-03-31T15:56:39.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.324+0000 7fab7e200640 1 -- start start 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.324+0000 7fab7e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab780d2b20 0x7fab780cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.324+0000 7fab7e200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab780af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab7e200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab780cefc0 con 0x7fab7815c320 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab7e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fab780aed70 con 0x7fab780d2b20 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab7e200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab780cf2a0 con 0x7fab780cd640 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab780d2b20 0x7fab780cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab77400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fab7815c320 0x7fab780cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38192/0 (socket says 172.21.15.90:38192) 2024-03-31T15:56:39.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fab780d2b20 0x7fab780cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59708/0 (socket says 172.21.15.90:59708) 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab77400640 1 -- 172.21.15.90:0/1599800493 learned_addr learned my addr 172.21.15.90:0/1599800493 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab77e00640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab780af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 -- 172.21.15.90:0/1599800493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 msgr2=0x7fab780af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab780af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 -- 172.21.15.90:0/1599800493 >> v1:172.21.15.90:6789/0 conn(0x7fab7815c320 legacy=0x7fab780cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 -- 172.21.15.90:0/1599800493 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab780cf580 con 0x7fab780d2b20 2024-03-31T15:56:39.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab76a00640 1 -- 172.21.15.90:0/1599800493 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223228247 0 0) 0x7fab780cefc0 con 0x7fab7815c320 2024-03-31T15:56:39.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab6fe00640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab780d2b20 0x7fab780cd100 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fab600097c0 tx=0x7fab6001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c610c123d447e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab76a00640 1 -- 172.21.15.90:0/1599800493 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600b0070 con 0x7fab780d2b20 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab76a00640 1 -- 172.21.15.90:0/1599800493 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab780d0f20 con 0x7fab780d2b20 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.328+0000 7fab76a00640 1 -- 172.21.15.90:0/1599800493 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab600ab820 con 0x7fab780d2b20 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- 172.21.15.90:0/1599800493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab780d2b20 msgr2=0x7fab780cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab780d2b20 0x7fab780cd100 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fab600097c0 tx=0x7fab6001c410 comp rx=0 tx=0).stop 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- 172.21.15.90:0/1599800493 shutdown_connections 2024-03-31T15:56:39.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab780af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- 172.21.15.90:0/1599800493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab780d2b20 0x7fab780cd100 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- 172.21.15.90:0/1599800493 >> 172.21.15.90:0/1599800493 conn(0x7fab780d1f20 msgr2=0x7fab780d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- 172.21.15.90:0/1599800493 shutdown_connections 2024-03-31T15:56:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- 172.21.15.90:0/1599800493 wait complete. 2024-03-31T15:56:39.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 Processor -- start 2024-03-31T15:56:39.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- start start 2024-03-31T15:56:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab781567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab780d2b20 0x7fab78154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fab780b0320 con 0x7fab7815c320 2024-03-31T15:56:39.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab780b01a0 con 0x7fab780d2b20 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab7e200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fab780b04a0 con 0x7fab780cd640 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59718/0 (socket says 172.21.15.90:59718) 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 -- 172.21.15.90:0/2540497616 learned_addr learned my addr 172.21.15.90:0/2540497616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77400640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab781567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab6fe00640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab780d2b20 0x7fab78154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 -- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 msgr2=0x7fab781567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab781567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 -- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab780d2b20 msgr2=0x7fab78154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab780d2b20 0x7fab78154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.332+0000 7fab77e00640 1 -- 172.21.15.90:0/2540497616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab78156e00 con 0x7fab7815c320 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab77e00640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fab70009fd0 tx=0x7fab7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70014420 con 0x7fab7815c320 2024-03-31T15:56:39.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab70019070 con 0x7fab7815c320 2024-03-31T15:56:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab781570e0 con 0x7fab7815c320 2024-03-31T15:56:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70014800 con 0x7fab7815c320 2024-03-31T15:56:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab781576a0 con 0x7fab7815c320 2024-03-31T15:56:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab780685f0 con 0x7fab7815c320 2024-03-31T15:56:39.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fab700149a0 con 0x7fab7815c320 2024-03-31T15:56:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 0x7fab5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.336+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fab70087dc0 con 0x7fab7815c320 2024-03-31T15:56:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.340+0000 7fab77400640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 0x7fab5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.340+0000 7fab77400640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 0x7fab5406a840 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fab68007fd0 tx=0x7fab68007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.340+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fab70057b80 con 0x7fab7815c320 2024-03-31T15:56:39.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.556+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) -- 0x7fab781666f0 con 0x7fab7815c320 2024-03-31T15:56:39.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.556+0000 7fab6f400640 1 -- 172.21.15.90:0/2540497616 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v18) ==== 183+0+140 (secure 0 0 0) 0x7fab7005be80 con 0x7fab7815c320 2024-03-31T15:56:39.551 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:56:39.551 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:56:39.551 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:56:39.552 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:56:39.552 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-31T15:56:39.552 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 msgr2=0x7fab5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 0x7fab5406a840 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fab68007fd0 tx=0x7fab68007d60 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 msgr2=0x7fab78158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fab70009fd0 tx=0x7fab7000e4b0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 shutdown_connections 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab54068380 0x7fab5406a840 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fab7815c320 0x7fab78158950 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab780d2b20 0x7fab78154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 --2- 172.21.15.90:0/2540497616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fab780cd640 0x7fab781567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.554 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 >> 172.21.15.90:0/2540497616 conn(0x7fab780d1f20 msgr2=0x7fab7806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:39.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 shutdown_connections 2024-03-31T15:56:39.555 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.560+0000 7fab7e200640 1 -- 172.21.15.90:0/2540497616 wait complete. 2024-03-31T15:56:39.573 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:39.573 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:39.582 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 Processor -- start 2024-03-31T15:56:39.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 -- start start 2024-03-31T15:56:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 0x7f299015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f29900b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29900b47e0 con 0x7f299006b820 2024-03-31T15:56:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f29900b4ac0 con 0x7f29901449a0 2024-03-31T15:56:39.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f2996600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f299015b7d0 con 0x7f29900b3d90 2024-03-31T15:56:39.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f298ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f29900b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f298fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f299006b820 0x7f299015ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:43352/0 (socket says 172.21.15.90:43352) 2024-03-31T15:56:39.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298fe00640 1 -- 172.21.15.90:0/645066365 learned_addr learned my addr 172.21.15.90:0/645066365 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:39.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.764+0000 7f298f400640 1 --2- 172.21.15.90:0/645066365 >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 0x7f299015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298ea00640 1 -- 172.21.15.90:0/645066365 >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 msgr2=0x7f299015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298ea00640 1 --2- 172.21.15.90:0/645066365 >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 0x7f299015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298ea00640 1 -- 172.21.15.90:0/645066365 >> v1:172.21.15.102:6789/0 conn(0x7f299006b820 legacy=0x7f299015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298e000640 1 -- 172.21.15.90:0/645066365 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1890486571 0 0) 0x7f29900b47e0 con 0x7f299006b820 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298ea00640 1 -- 172.21.15.90:0/645066365 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299015bab0 con 0x7f29900b3d90 2024-03-31T15:56:39.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298f400640 1 --2- 172.21.15.90:0/645066365 >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 0x7f299015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:39.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.768+0000 7f298ea00640 1 --2- 172.21.15.90:0/645066365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f29900b41f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ba60 tx=0x7f299015f970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949d45dc70386b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298e000640 1 -- 172.21.15.90:0/645066365 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c0b4070 con 0x7f29900b3d90 2024-03-31T15:56:39.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298e000640 1 -- 172.21.15.90:0/645066365 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f297c0a7ba0 con 0x7f29900b3d90 2024-03-31T15:56:39.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298e000640 1 -- 172.21.15.90:0/645066365 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c0af440 con 0x7f29900b3d90 2024-03-31T15:56:39.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- 172.21.15.90:0/645066365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 msgr2=0x7f29900b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- 172.21.15.90:0/645066365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f29900b41f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ba60 tx=0x7f299015f970 comp rx=0 tx=0).stop 2024-03-31T15:56:39.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- 172.21.15.90:0/645066365 shutdown_connections 2024-03-31T15:56:39.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- 172.21.15.90:0/645066365 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f29900b41f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- 172.21.15.90:0/645066365 >> v2:172.21.15.102:3300/0 conn(0x7f29901449a0 0x7f299015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- 172.21.15.90:0/645066365 >> 172.21.15.90:0/645066365 conn(0x7f29900b1c80 msgr2=0x7f29900b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:39.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- 172.21.15.90:0/645066365 shutdown_connections 2024-03-31T15:56:39.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- 172.21.15.90:0/645066365 wait complete. 2024-03-31T15:56:39.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 Processor -- start 2024-03-31T15:56:39.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- start start 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f299013d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 0x7f29900ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f29900b2e00 con 0x7f299006b820 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f29900b2c80 con 0x7f29901449a0 2024-03-31T15:56:39.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f2996600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f29900b2f80 con 0x7f29900b3d90 2024-03-31T15:56:39.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298f400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59730/0 (socket says 172.21.15.90:59730) 2024-03-31T15:56:39.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.772+0000 7f298f400640 1 -- 172.21.15.90:0/3881358907 learned_addr learned my addr 172.21.15.90:0/3881358907 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298ea00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f299013d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 -- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 msgr2=0x7f299013d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298fe00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 0x7f29900ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f299013d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 -- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 msgr2=0x7f29900ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 0x7f29900ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 -- 172.21.15.90:0/3881358907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299013dd90 con 0x7f299006b820 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298ea00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f299013d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:39.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298fe00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 0x7f29900ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f298f400640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f298000e980 tx=0x7f298000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980017070 con 0x7f299006b820 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298000ca00 con 0x7f299006b820 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298000cdd0 con 0x7f299006b820 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299013e070 con 0x7f299006b820 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2990138b70 con 0x7f299006b820 2024-03-31T15:56:39.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299013e590 con 0x7f299006b820 2024-03-31T15:56:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f298001a070 con 0x7f299006b820 2024-03-31T15:56:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 0x7f297006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.776+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f2980086e60 con 0x7f299006b820 2024-03-31T15:56:39.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.780+0000 7f298ea00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 0x7f297006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:39.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.780+0000 7f298ea00640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 0x7f297006a840 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f297c0b1000 tx=0x7f297c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:39.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.780+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f299013e590 con 0x7f299006b820 2024-03-31T15:56:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.940+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29900686d0 con 0x7f299006b820 2024-03-31T15:56:39.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2987400640 1 -- 172.21.15.90:0/3881358907 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f29800571f0 con 0x7f299006b820 2024-03-31T15:56:39.937 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:39.937 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 msgr2=0x7f297006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 0x7f297006a840 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f297c0b1000 tx=0x7f297c008040 comp rx=0 tx=0).stop 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 msgr2=0x7f2990138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f298000e980 tx=0x7f298000ee40 comp rx=0 tx=0).stop 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 shutdown_connections 2024-03-31T15:56:39.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2970068380 0x7f297006a840 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29901449a0 0x7f29900ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29900b3d90 0x7f299013d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 --2- 172.21.15.90:0/3881358907 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f299006b820 0x7f2990138630 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 >> 172.21.15.90:0/3881358907 conn(0x7f29900b1c80 msgr2=0x7f29900b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 shutdown_connections 2024-03-31T15:56:39.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:39.944+0000 7f2996600640 1 -- 172.21.15.90:0/3881358907 wait complete. 2024-03-31T15:56:39.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:39.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:40.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 Processor -- start 2024-03-31T15:56:40.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 -- start start 2024-03-31T15:56:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea406d020 0x7f3ea40ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ea406d460 0x7f3ea40cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea415e8c0 con 0x7f3ea415e4e0 2024-03-31T15:56:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea40d0ff0 con 0x7f3ea406d460 2024-03-31T15:56:40.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3eaa600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea40d12d0 con 0x7f3ea406d020 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ea406d460 0x7f3ea40cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3ea415e4e0 0x7f3ea406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:43368/0 (socket says 172.21.15.90:43368) 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3ea406d460 0x7f3ea40cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.90:59380/0 (socket says 172.21.15.90:59380) 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea2a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea406d020 0x7f3ea40ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3400640 1 -- 172.21.15.90:0/304039027 learned_addr learned my addr 172.21.15.90:0/304039027 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 -- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea406d020 msgr2=0x7f3ea40ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:40.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 --2- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea406d020 0x7f3ea40ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 -- 172.21.15.90:0/304039027 >> v1:172.21.15.102:6789/0 conn(0x7f3ea415e4e0 legacy=0x7f3ea406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea2000640 1 -- 172.21.15.90:0/304039027 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4167297279 0 0) 0x7f3ea415e8c0 con 0x7f3ea415e4e0 2024-03-31T15:56:40.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.108+0000 7f3ea3e00640 1 -- 172.21.15.90:0/304039027 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea40cc710 con 0x7f3ea406d460 2024-03-31T15:56:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3ea3e00640 1 --2- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40cd450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009d90 tx=0x7f3e9c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4a9447b76fc586a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3ea2000640 1 -- 172.21.15.90:0/304039027 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c0b2070 con 0x7f3ea406d460 2024-03-31T15:56:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3ea2000640 1 -- 172.21.15.90:0/304039027 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e9c0a57a0 con 0x7f3ea406d460 2024-03-31T15:56:40.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3ea2000640 1 -- 172.21.15.90:0/304039027 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c0a5b70 con 0x7f3ea406d460 2024-03-31T15:56:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 -- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 msgr2=0x7f3ea40cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 --2- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40cd450 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c009d90 tx=0x7f3e9c01c8e0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 -- 172.21.15.90:0/304039027 shutdown_connections 2024-03-31T15:56:40.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 --2- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40cd450 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 --2- 172.21.15.90:0/304039027 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea406d020 0x7f3ea40ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 -- 172.21.15.90:0/304039027 >> 172.21.15.90:0/304039027 conn(0x7f3ea40d1f20 msgr2=0x7f3ea40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 -- 172.21.15.90:0/304039027 shutdown_connections 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.112+0000 7f3eaa600640 1 -- 172.21.15.90:0/304039027 wait complete. 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 Processor -- start 2024-03-31T15:56:40.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- start start 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea415e4e0 0x7f3ea40c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea40d0d90 con 0x7f3ea406d020 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea40d0c10 con 0x7f3ea406d460 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea416b030 con 0x7f3ea415e4e0 2024-03-31T15:56:40.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59744/0 (socket says 172.21.15.90:59744) 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 -- 172.21.15.90:0/46240925 learned_addr learned my addr 172.21.15.90:0/46240925 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3e00640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea415e4e0 0x7f3ea40c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea2a00640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 -- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea415e4e0 msgr2=0x7f3ea40c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea415e4e0 0x7f3ea40c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 -- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 msgr2=0x7f3ea40c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 -- 172.21.15.90:0/46240925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea40ca090 con 0x7f3ea406d020 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3ea3400640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400c480 tx=0x7f3e9400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94018070 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e94007500 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94004a30 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea40c7960 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.116+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea40c7f20 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea40685f0 con 0x7f3ea406d020 2024-03-31T15:56:40.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3e94004bd0 con 0x7f3ea406d020 2024-03-31T15:56:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3e9b400640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 0x7f3e8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f3e94085d00 con 0x7f3ea406d020 2024-03-31T15:56:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3ea2a00640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 0x7f3e8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.120+0000 7f3ea2a00640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 0x7f3e8406a840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ea415b280 tx=0x7f3e90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.124+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3e94055ad0 con 0x7f3ea406d020 2024-03-31T15:56:40.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.284+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ea40b7980 con 0x7f3ea406d020 2024-03-31T15:56:40.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.284+0000 7f3e9b400640 1 -- 172.21.15.90:0/46240925 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f3e94059dd0 con 0x7f3ea406d020 2024-03-31T15:56:40.279 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 msgr2=0x7f3e8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 0x7f3e8406a840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ea415b280 tx=0x7f3e90009190 comp rx=0 tx=0).stop 2024-03-31T15:56:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 msgr2=0x7f3ea4149870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400c480 tx=0x7f3e9400c940 comp rx=0 tx=0).stop 2024-03-31T15:56:40.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 shutdown_connections 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3e84068380 0x7f3e8406a840 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3ea415e4e0 0x7f3ea40c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ea406d460 0x7f3ea40c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 --2- 172.21.15.90:0/46240925 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ea406d020 0x7f3ea4149870 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 >> 172.21.15.90:0/46240925 conn(0x7f3ea40d1f20 msgr2=0x7f3ea40708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 shutdown_connections 2024-03-31T15:56:40.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.288+0000 7f3eaa600640 1 -- 172.21.15.90:0/46240925 wait complete. 2024-03-31T15:56:40.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:40.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.444+0000 7f0111200640 1 Processor -- start 2024-03-31T15:56:40.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0111200640 1 -- start start 2024-03-31T15:56:40.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0111200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f010c06d020 0x7f010c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0111200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c15e8c0 con 0x7f010c06d490 2024-03-31T15:56:40.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0111200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f010c0d0ff0 con 0x7f010c15e4e0 2024-03-31T15:56:40.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0111200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f010c0d12d0 con 0x7f010c06d020 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f010a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f010c06d490 0x7f010c0cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38208/0 (socket says 172.21.15.90:38208) 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f010a800640 1 -- 172.21.15.90:0/2731525683 learned_addr learned my addr 172.21.15.90:0/2731525683 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0103e00640 1 --2- 172.21.15.90:0/2731525683 >> v2:172.21.15.102:3300/0 conn(0x7f010c06d020 0x7f010c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1906847190 0 0) 0x7f010c15e8c0 con 0x7f010c06d490 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00f0003530 con 0x7f010c06d490 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980318536 0 0) 0x7f010c0d0ff0 con 0x7f010c15e4e0 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f010c15e8c0 con 0x7f010c15e4e0 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0103e00640 1 -- 172.21.15.90:0/2731525683 >> v1:172.21.15.102:6789/0 conn(0x7f010c15e4e0 legacy=0x7f010c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0103e00640 1 -- 172.21.15.90:0/2731525683 >> v1:172.21.15.90:6789/0 conn(0x7f010c06d490 legacy=0x7f010c0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.448+0000 7f0103e00640 1 -- 172.21.15.90:0/2731525683 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c0cc710 con 0x7f010c06d020 2024-03-31T15:56:40.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0103e00640 1 --2- 172.21.15.90:0/2731525683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d020 0x7f010c0ccf10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f00f40097f0 tx=0x7f00f401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47242e1c940f394f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f40b0070 con 0x7f010c06d020 2024-03-31T15:56:40.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f40ab600 con 0x7f010c06d020 2024-03-31T15:56:40.445 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0109400640 1 -- 172.21.15.90:0/2731525683 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f40ab9b0 con 0x7f010c06d020 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 -- 172.21.15.90:0/2731525683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d020 msgr2=0x7f010c0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 --2- 172.21.15.90:0/2731525683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d020 0x7f010c0ccf10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f00f40097f0 tx=0x7f00f401c410 comp rx=0 tx=0).stop 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 -- 172.21.15.90:0/2731525683 shutdown_connections 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 --2- 172.21.15.90:0/2731525683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d020 0x7f010c0ccf10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 -- 172.21.15.90:0/2731525683 >> 172.21.15.90:0/2731525683 conn(0x7f010c0d1f20 msgr2=0x7f010c0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 -- 172.21.15.90:0/2731525683 shutdown_connections 2024-03-31T15:56:40.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.452+0000 7f0111200640 1 -- 172.21.15.90:0/2731525683 wait complete. 2024-03-31T15:56:40.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 Processor -- start 2024-03-31T15:56:40.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 -- start start 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d490 0x7f010c152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f010c15e4e0 0x7f010c0cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f010c0d0d90 con 0x7f010c06d020 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f010c0d0c10 con 0x7f010c06d490 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f010c16c040 con 0x7f010c15e4e0 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59750/0 (socket says 172.21.15.90:59750) 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 -- 172.21.15.90:0/1021267325 learned_addr learned my addr 172.21.15.90:0/1021267325 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0103e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d490 0x7f010c152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f010a800640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f010c15e4e0 0x7f010c0cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 -- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f010c15e4e0 msgr2=0x7f010c0cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.452 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f010c15e4e0 0x7f010c0cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 -- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d490 msgr2=0x7f010c152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d490 0x7f010c152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 -- 172.21.15.90:0/1021267325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c0cb820 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0109e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00c500 tx=0x7f00fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0052d0 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00fc005490 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c0c9aa0 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.456+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc012850 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c0ca060 con 0x7f010c06d020 2024-03-31T15:56:40.453 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c0685f0 con 0x7f010c06d020 2024-03-31T15:56:40.454 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f00fc0129f0 con 0x7f010c06d020 2024-03-31T15:56:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0102000640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 0x7f00e806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f00fc086810 con 0x7f010c06d020 2024-03-31T15:56:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.460+0000 7f0103e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 0x7f00e806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.455 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.464+0000 7f0103e00640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 0x7f00e806a840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00f40ad000 tx=0x7f00f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.458 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.464+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f00fc056bb0 con 0x7f010c06d020 2024-03-31T15:56:40.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.640+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f010c0cba50 con 0x7f010c06d020 2024-03-31T15:56:40.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.640+0000 7f0102000640 1 -- 172.21.15.90:0/1021267325 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7f00fc0564b0 con 0x7f010c06d020 2024-03-31T15:56:40.635 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:40.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 msgr2=0x7f00e806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 0x7f00e806a840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00f40ad000 tx=0x7f00f4008040 comp rx=0 tx=0).stop 2024-03-31T15:56:40.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 msgr2=0x7f010c1553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f00fc00c500 tx=0x7f00fc00c9c0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 shutdown_connections 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f00e8068380 0x7f00e806a840 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f010c15e4e0 0x7f010c0cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f010c06d490 0x7f010c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 --2- 172.21.15.90:0/1021267325 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f010c06d020 0x7f010c1553c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 >> 172.21.15.90:0/1021267325 conn(0x7f010c0d1f20 msgr2=0x7f010c0c5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 shutdown_connections 2024-03-31T15:56:40.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.644+0000 7f0111200640 1 -- 172.21.15.90:0/1021267325 wait complete. 2024-03-31T15:56:40.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 Processor -- start 2024-03-31T15:56:40.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 -- start start 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2580d2b20 0x7fc2580cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2580cd640 0x7fc2580af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2580cefc0 con 0x7fc25815c320 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc2580aed70 con 0x7fc2580cd640 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc25ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc2580cf2a0 con 0x7fc2580d2b20 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc255e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc25815c320 0x7fc2580cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38210/0 (socket says 172.21.15.90:38210) 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc256800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc2580cd640 0x7fc2580af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc255e00640 1 -- 172.21.15.90:0/934827780 learned_addr learned my addr 172.21.15.90:0/934827780 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc24fe00640 1 --2- 172.21.15.90:0/934827780 >> v2:172.21.15.102:3300/0 conn(0x7fc2580d2b20 0x7fc2580cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc256800640 1 -- 172.21.15.90:0/934827780 >> v2:172.21.15.102:3300/0 conn(0x7fc2580d2b20 msgr2=0x7fc2580cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc256800640 1 --2- 172.21.15.90:0/934827780 >> v2:172.21.15.102:3300/0 conn(0x7fc2580d2b20 0x7fc2580cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc256800640 1 -- 172.21.15.90:0/934827780 >> v1:172.21.15.90:6789/0 conn(0x7fc25815c320 legacy=0x7fc2580cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc256800640 1 -- 172.21.15.90:0/934827780 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2580cf580 con 0x7fc2580cd640 2024-03-31T15:56:40.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.804+0000 7fc255400640 1 -- 172.21.15.90:0/934827780 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353260315 0 0) 0x7fc2580cefc0 con 0x7fc25815c320 2024-03-31T15:56:40.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc256800640 1 --2- 172.21.15.90:0/934827780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc2580af200 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc2580b7fd0 tx=0x7fc25001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c27910802a8bb7f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc255400640 1 -- 172.21.15.90:0/934827780 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500b2070 con 0x7fc2580cd640 2024-03-31T15:56:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc255400640 1 -- 172.21.15.90:0/934827780 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2580d0f20 con 0x7fc2580cd640 2024-03-31T15:56:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc255400640 1 -- 172.21.15.90:0/934827780 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500a5a90 con 0x7fc2580cd640 2024-03-31T15:56:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 -- 172.21.15.90:0/934827780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 msgr2=0x7fc2580af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.802 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 --2- 172.21.15.90:0/934827780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc2580af200 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc2580b7fd0 tx=0x7fc25001c8a0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 -- 172.21.15.90:0/934827780 shutdown_connections 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 --2- 172.21.15.90:0/934827780 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc2580af200 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 --2- 172.21.15.90:0/934827780 >> v2:172.21.15.102:3300/0 conn(0x7fc2580d2b20 0x7fc2580cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 -- 172.21.15.90:0/934827780 >> 172.21.15.90:0/934827780 conn(0x7fc2580d1f20 msgr2=0x7fc2580d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 -- 172.21.15.90:0/934827780 shutdown_connections 2024-03-31T15:56:40.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.808+0000 7fc25ca00640 1 -- 172.21.15.90:0/934827780 wait complete. 2024-03-31T15:56:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 Processor -- start 2024-03-31T15:56:40.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 -- start start 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2580d2b20 0x7fc2581fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc25815c320 0x7fc258155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2580b02e0 con 0x7fc2580cd640 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc2580b0160 con 0x7fc2580d2b20 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc25ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc2580b0460 con 0x7fc25815c320 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59768/0 (socket says 172.21.15.90:59768) 2024-03-31T15:56:40.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 -- 172.21.15.90:0/794860381 learned_addr learned my addr 172.21.15.90:0/794860381 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc256800640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc25815c320 0x7fc258155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc24fe00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2580d2b20 0x7fc2581fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 -- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc25815c320 msgr2=0x7fc258155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc25815c320 0x7fc258155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 -- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2580d2b20 msgr2=0x7fc2581fb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2580d2b20 0x7fc2581fb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.812+0000 7fc255e00640 1 -- 172.21.15.90:0/794860381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2581faae0 con 0x7fc2580cd640 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc255e00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c520 tx=0x7fc24800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248018070 con 0x7fc2580cd640 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2480074e0 con 0x7fc2580cd640 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2480049f0 con 0x7fc2580cd640 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25815d0c0 con 0x7fc2580cd640 2024-03-31T15:56:40.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2581fad80 con 0x7fc2580cd640 2024-03-31T15:56:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2580685f0 con 0x7fc2580cd640 2024-03-31T15:56:40.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc24801b070 con 0x7fc2580cd640 2024-03-31T15:56:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 0x7fc23406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fc248085ef0 con 0x7fc2580cd640 2024-03-31T15:56:40.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.816+0000 7fc24fe00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 0x7fc23406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:40.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.820+0000 7fc24fe00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 0x7fc23406a840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc25815ab60 tx=0x7fc240008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:40.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.820+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc2480563b0 con 0x7fc2580cd640 2024-03-31T15:56:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.988+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2580c8940 con 0x7fc2580cd640 2024-03-31T15:56:40.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.988+0000 7fc24e000640 1 -- 172.21.15.90:0/794860381 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7fc248055cb0 con 0x7fc2580cd640 2024-03-31T15:56:40.983 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 msgr2=0x7fc23406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 0x7fc23406a840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc25815ab60 tx=0x7fc240008040 comp rx=0 tx=0).stop 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 msgr2=0x7fc25815c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc24800c520 tx=0x7fc24800c9e0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 shutdown_connections 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc234068380 0x7fc23406a840 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc25815c320 0x7fc258155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2580d2b20 0x7fc2581fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 --2- 172.21.15.90:0/794860381 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2580cd640 0x7fc25815c970 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:40.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 >> 172.21.15.90:0/794860381 conn(0x7fc2580d1f20 msgr2=0x7fc258070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:40.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 shutdown_connections 2024-03-31T15:56:40.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:40.992+0000 7fc25ca00640 1 -- 172.21.15.90:0/794860381 wait complete. 2024-03-31T15:56:41.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:41.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 Processor -- start 2024-03-31T15:56:41.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 -- start start 2024-03-31T15:56:41.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd9480ce910 0x7fd9480cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd9480ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9480cf230 con 0x7fd9480cfe90 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fd9480cd460 con 0x7fd9480ce910 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd94ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd9480cd740 con 0x7fd94815d240 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd9480ce910 0x7fd9480cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fd9480ce910 0x7fd9480cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59780/0 (socket says 172.21.15.90:59780) 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 -- 172.21.15.90:0/900008602 learned_addr learned my addr 172.21.15.90:0/900008602 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945400640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd9480ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 -- 172.21.15.90:0/900008602 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 msgr2=0x7fd9480ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd9480ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 -- 172.21.15.90:0/900008602 >> v1:172.21.15.102:6789/0 conn(0x7fd9480cfe90 legacy=0x7fd9480d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.152+0000 7fd945e00640 1 -- 172.21.15.90:0/900008602 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9480cda20 con 0x7fd9480ce910 2024-03-31T15:56:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd945e00640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd9480cecf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd938009870 tx=0x7fd93801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c86ce0a3aa1c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd944a00640 1 -- 172.21.15.90:0/900008602 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9380b1070 con 0x7fd9480ce910 2024-03-31T15:56:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd944a00640 1 -- 172.21.15.90:0/900008602 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9380ac410 con 0x7fd9480ce910 2024-03-31T15:56:41.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd944a00640 1 -- 172.21.15.90:0/900008602 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9380ac7e0 con 0x7fd9480ce910 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- 172.21.15.90:0/900008602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 msgr2=0x7fd9480cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd9480cecf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd938009870 tx=0x7fd93801c4a0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- 172.21.15.90:0/900008602 shutdown_connections 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd9480ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 --2- 172.21.15.90:0/900008602 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd9480cecf0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- 172.21.15.90:0/900008602 >> 172.21.15.90:0/900008602 conn(0x7fd94808bfd0 msgr2=0x7fd9480d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- 172.21.15.90:0/900008602 shutdown_connections 2024-03-31T15:56:41.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- 172.21.15.90:0/900008602 wait complete. 2024-03-31T15:56:41.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 Processor -- start 2024-03-31T15:56:41.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.156+0000 7fd94ca00640 1 -- start start 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9480cfe90 0x7fd948155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd948156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd9480cd1d0 con 0x7fd9480ce910 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd9480cd050 con 0x7fd9480cfe90 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd948169410 con 0x7fd94815d240 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59794/0 (socket says 172.21.15.90:59794) 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 -- 172.21.15.90:0/293041772 learned_addr learned my addr 172.21.15.90:0/293041772 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945400640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9480cfe90 0x7fd948155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd946800640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd948156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 -- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 msgr2=0x7fd948156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd948156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 -- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9480cfe90 msgr2=0x7fd948155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9480cfe90 0x7fd948155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 -- 172.21.15.90:0/293041772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94814eec0 con 0x7fd9480ce910 2024-03-31T15:56:41.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd945e00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd93801a0d0 tx=0x7fd938019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9380b1040 con 0x7fd9480ce910 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9380ba070 con 0x7fd9480ce910 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9380b5440 con 0x7fd9480ce910 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94814f1a0 con 0x7fd9480ce910 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9480c54d0 con 0x7fd9480ce910 2024-03-31T15:56:41.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd948068660 con 0x7fd9480ce910 2024-03-31T15:56:41.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.160+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9380b55e0 con 0x7fd9480ce910 2024-03-31T15:56:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.164+0000 7fd93e000640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 0x7fd92806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.164+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fd938120630 con 0x7fd9480ce910 2024-03-31T15:56:41.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.164+0000 7fd945400640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 0x7fd92806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.158 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.164+0000 7fd945400640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 0x7fd92806a870 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b7f30 tx=0x7fd934008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.168+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd938016150 con 0x7fd9480ce910 2024-03-31T15:56:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.356+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd9480ce6f0 con 0x7fd9480ce910 2024-03-31T15:56:41.351 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.356+0000 7fd93e000640 1 -- 172.21.15.90:0/293041772 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7fd9380f09c0 con 0x7fd9480ce910 2024-03-31T15:56:41.354 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 msgr2=0x7fd92806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 0x7fd92806a870 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b7f30 tx=0x7fd934008040 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 msgr2=0x7fd948166520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd93801a0d0 tx=0x7fd938019240 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 shutdown_connections 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd9280683b0 0x7fd92806a870 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd94815d240 0x7fd948156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9480cfe90 0x7fd948155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 --2- 172.21.15.90:0/293041772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd9480ce910 0x7fd948166520 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 >> 172.21.15.90:0/293041772 conn(0x7fd94808bfd0 msgr2=0x7fd94806f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 shutdown_connections 2024-03-31T15:56:41.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.360+0000 7fd94ca00640 1 -- 172.21.15.90:0/293041772 wait complete. 2024-03-31T15:56:41.373 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:41.373 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:41.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:41.557 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 Processor -- start 2024-03-31T15:56:41.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 -- start start 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe05415c120 0x7fe05415fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 0x7fe0540b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0540d2b10 con 0x7fe0540b2c20 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fe054160bf0 con 0x7fe05415c120 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe05c400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe054160350 con 0x7fe0540d2680 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe05415c120 0x7fe05415fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fe05415c120 0x7fe05415fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59808/0 (socket says 172.21.15.90:59808) 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 -- 172.21.15.90:0/2090355136 learned_addr learned my addr 172.21.15.90:0/2090355136 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059000640 1 --2- 172.21.15.90:0/2090355136 >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 0x7fe0540b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 -- 172.21.15.90:0/2090355136 >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 msgr2=0x7fe0540b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 --2- 172.21.15.90:0/2090355136 >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 0x7fe0540b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 -- 172.21.15.90:0/2090355136 >> v1:172.21.15.90:6789/0 conn(0x7fe0540b2c20 legacy=0x7fe0540cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059a00640 1 -- 172.21.15.90:0/2090355136 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe054160630 con 0x7fe05415c120 2024-03-31T15:56:41.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.564+0000 7fe059000640 1 --2- 172.21.15.90:0/2090355136 >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 0x7fe0540b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:41.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe059a00640 1 --2- 172.21.15.90:0/2090355136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe05415c120 0x7fe05415fd00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe05406ed50 tx=0x7fe04801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272d168be592d3f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe053e00640 1 -- 172.21.15.90:0/2090355136 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0480ab660 con 0x7fe05415c120 2024-03-31T15:56:41.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe053e00640 1 -- 172.21.15.90:0/2090355136 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0480b0070 con 0x7fe05415c120 2024-03-31T15:56:41.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe053e00640 1 -- 172.21.15.90:0/2090355136 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0480aba20 con 0x7fe05415c120 2024-03-31T15:56:41.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 -- 172.21.15.90:0/2090355136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe05415c120 msgr2=0x7fe05415fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 --2- 172.21.15.90:0/2090355136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe05415c120 0x7fe05415fd00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe05406ed50 tx=0x7fe04801c460 comp rx=0 tx=0).stop 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 -- 172.21.15.90:0/2090355136 shutdown_connections 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 --2- 172.21.15.90:0/2090355136 >> v2:172.21.15.102:3300/0 conn(0x7fe0540d2680 0x7fe0540b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 --2- 172.21.15.90:0/2090355136 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe05415c120 0x7fe05415fd00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 -- 172.21.15.90:0/2090355136 >> 172.21.15.90:0/2090355136 conn(0x7fe05409aa40 msgr2=0x7fe05409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 -- 172.21.15.90:0/2090355136 shutdown_connections 2024-03-31T15:56:41.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.568+0000 7fe05c400640 1 -- 172.21.15.90:0/2090355136 wait complete. 2024-03-31T15:56:41.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 Processor -- start 2024-03-31T15:56:41.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- start start 2024-03-31T15:56:41.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 0x7fe0541fba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe05415c120 0x7fe0541f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe0541691c0 con 0x7fe0540d2680 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe054169040 con 0x7fe05415c120 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe054169340 con 0x7fe0540b2c20 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59820/0 (socket says 172.21.15.90:59820) 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 -- 172.21.15.90:0/1037812882 learned_addr learned my addr 172.21.15.90:0/1037812882 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05a400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe05415c120 0x7fe0541f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059a00640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 0x7fe0541fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 -- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 msgr2=0x7fe0541fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 0x7fe0541fba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 -- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe05415c120 msgr2=0x7fe0541f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe05415c120 0x7fe0541f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 -- 172.21.15.90:0/1037812882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0541f81d0 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059a00640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 0x7fe0541fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe059000640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c820 tx=0x7fe04000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040015070 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe040007a30 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040007e40 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0541f84b0 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.572+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe05406cb20 con 0x7fe0540d2680 2024-03-31T15:56:41.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.576+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05406c910 con 0x7fe0540d2680 2024-03-31T15:56:41.570 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.576+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe040018070 con 0x7fe0540d2680 2024-03-31T15:56:41.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.576+0000 7fe051600640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 0x7fe03406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.576+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fe040086a30 con 0x7fe0540d2680 2024-03-31T15:56:41.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.580+0000 7fe059a00640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 0x7fe03406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.580+0000 7fe059a00640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 0x7fe03406a840 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe0480af000 tx=0x7fe048008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.584+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe040056e60 con 0x7fe0540d2680 2024-03-31T15:56:41.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.740+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0540686d0 con 0x7fe0540d2680 2024-03-31T15:56:41.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe051600640 1 -- 172.21.15.90:0/1037812882 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7fe040056760 con 0x7fe0540d2680 2024-03-31T15:56:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:41.736 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 msgr2=0x7fe03406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 0x7fe03406a840 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe0480af000 tx=0x7fe048008040 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 msgr2=0x7fe054159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c820 tx=0x7fe04000cce0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 shutdown_connections 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe034068380 0x7fe03406a840 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe05415c120 0x7fe0541f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0540d2680 0x7fe054159f60 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 --2- 172.21.15.90:0/1037812882 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe0540b2c20 0x7fe0541fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 >> 172.21.15.90:0/1037812882 conn(0x7fe05409aa40 msgr2=0x7fe05409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:41.739 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 shutdown_connections 2024-03-31T15:56:41.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.744+0000 7fe05c400640 1 -- 172.21.15.90:0/1037812882 wait complete. 2024-03-31T15:56:41.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:41.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:41.897 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 Processor -- start 2024-03-31T15:56:41.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 -- start start 2024-03-31T15:56:41.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f180b0210 0x7f3f18166eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f180cefc0 con 0x7f3f1815c320 2024-03-31T15:56:41.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f180cf2a0 con 0x7f3f180d2b20 2024-03-31T15:56:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f1d600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3f180aed70 con 0x7f3f180b0210 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f16a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f3f1815c320 0x7f3f180cc9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38230/0 (socket says 172.21.15.90:38230) 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f16a00640 1 -- 172.21.15.90:0/1474693042 learned_addr learned my addr 172.21.15.90:0/1474693042 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f17400640 1 --2- 172.21.15.90:0/1474693042 >> v2:172.21.15.102:3300/0 conn(0x7f3f180b0210 0x7f3f18166eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1458966958 0 0) 0x7f3f180cefc0 con 0x7f3f1815c320 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.904+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3efc003530 con 0x7f3f1815c320 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1295742212 0 0) 0x7f3efc003530 con 0x7f3f1815c320 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 >> v2:172.21.15.102:3300/0 conn(0x7f3f180b0210 msgr2=0x7f3f18166eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 --2- 172.21.15.90:0/1474693042 >> v2:172.21.15.102:3300/0 conn(0x7f3f180b0210 0x7f3f18166eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 >> v1:172.21.15.102:6789/0 conn(0x7f3f180d2b20 legacy=0x7f3f180cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f180cf580 con 0x7f3f1815c320 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f3f08001680 con 0x7f3f1815c320 2024-03-31T15:56:41.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 >> v1:172.21.15.90:6789/0 conn(0x7f3f1815c320 legacy=0x7f3f180cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3efc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3efc0043c0 0x7f3efc004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3efc005130 0x7f3efc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3f180aed70 con 0x7f3efc0039c0 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc00c070 con 0x7f3efc005130 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc00c270 con 0x7f3efc0043c0 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3efc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f17400640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3efc005130 0x7f3efc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f0fe00640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3efc0043c0 0x7f3efc004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 -- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3efc0043c0 msgr2=0x7f3efc004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3efc0043c0 0x7f3efc004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 -- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3efc005130 msgr2=0x7f3efc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3efc005130 0x7f3efc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 -- 172.21.15.90:0/1474693042 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc003530 con 0x7f3efc0039c0 2024-03-31T15:56:41.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f17400640 1 -- 172.21.15.90:0/1474693042 reap_dead start 2024-03-31T15:56:41.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.908+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3efc003da0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c190 tx=0x7f3f0801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab8f9600117f8a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08007cd0 con 0x7f3efc0039c0 2024-03-31T15:56:41.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f080a9070 con 0x7f3efc0039c0 2024-03-31T15:56:41.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f16000640 1 -- 172.21.15.90:0/1474693042 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f08005d60 con 0x7f3efc0039c0 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 msgr2=0x7f3efc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3efc003da0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800c190 tx=0x7f3f0801cba0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- 172.21.15.90:0/1474693042 shutdown_connections 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1474693042 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3efc003da0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- 172.21.15.90:0/1474693042 >> 172.21.15.90:0/1474693042 conn(0x7f3f180d1f20 msgr2=0x7f3f180d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- 172.21.15.90:0/1474693042 shutdown_connections 2024-03-31T15:56:41.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- 172.21.15.90:0/1474693042 wait complete. 2024-03-31T15:56:41.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 Processor -- start 2024-03-31T15:56:41.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 -- start start 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 0x7f3f18158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.912+0000 7f3f1d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18159010 0x7f3f1806d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3f180cd4c0 con 0x7f3efc0039c0 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f180cd340 con 0x7f3f18159010 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f3f180cd640 con 0x7f3f1815c970 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f16a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59836/0 (socket says 172.21.15.90:59836) 2024-03-31T15:56:41.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f16a00640 1 -- 172.21.15.90:0/1825246030 learned_addr learned my addr 172.21.15.90:0/1825246030 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 0x7f3f18158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f17400640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18159010 0x7f3f1806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 -- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18159010 msgr2=0x7f3f1806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18159010 0x7f3f1806d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 -- 172.21.15.90:0/1825246030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 msgr2=0x7f3f18154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 -- 172.21.15.90:0/1825246030 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f18154980 con 0x7f3f1815c970 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0fe00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 0x7f3f18158ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1806a3e0 tx=0x7f3f0000fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0001a070 con 0x7f3f1815c970 2024-03-31T15:56:41.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f00005110 con 0x7f3f1815c970 2024-03-31T15:56:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00007050 con 0x7f3f1815c970 2024-03-31T15:56:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f180ca070 con 0x7f3f1815c970 2024-03-31T15:56:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f180ca630 con 0x7f3f1815c970 2024-03-31T15:56:41.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.916+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f180685f0 con 0x7f3f1815c970 2024-03-31T15:56:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.920+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3f000071f0 con 0x7f3f1815c970 2024-03-31T15:56:41.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.920+0000 7f3f0ea00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 0x7f3ef406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:41.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.920+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f3f000860a0 con 0x7f3f1815c970 2024-03-31T15:56:41.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.920+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 0x7f3ef406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:41.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.920+0000 7f3f16a00640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 0x7f3ef406a840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f08000c10 tx=0x7f3f080a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:41.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:41.924+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f3f000074e0 con 0x7f3f1815c970 2024-03-31T15:56:42.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.088+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f180752b0 con 0x7f3f1815c970 2024-03-31T15:56:42.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.088+0000 7f3f0ea00640 1 -- 172.21.15.90:0/1825246030 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f3f00055e00 con 0x7f3f1815c970 2024-03-31T15:56:42.082 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:42.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 msgr2=0x7f3ef406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 0x7f3ef406a840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f08000c10 tx=0x7f3f080a7070 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 msgr2=0x7f3f18158ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 0x7f3f18158ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f1806a3e0 tx=0x7f3f0000fc00 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 shutdown_connections 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f3ef4068380 0x7f3ef406a840 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f18159010 0x7f3f1806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f3f1815c970 0x7f3f18158ad0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 --2- 172.21.15.90:0/1825246030 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3efc0039c0 0x7f3f18154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 >> 172.21.15.90:0/1825246030 conn(0x7f3f180d1f20 msgr2=0x7f3f180d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 shutdown_connections 2024-03-31T15:56:42.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.092+0000 7f3f1d600640 1 -- 172.21.15.90:0/1825246030 wait complete. 2024-03-31T15:56:42.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 Processor -- start 2024-03-31T15:56:42.244 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 -- start start 2024-03-31T15:56:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f880c1619c0 0x7f880c0d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880c0705c0 con 0x7f880c161fc0 2024-03-31T15:56:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880c0cff60 con 0x7f880c070020 2024-03-31T15:56:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880b200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f880c161da0 con 0x7f880c1619c0 2024-03-31T15:56:42.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f8809e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f880c1619c0 0x7f880c0d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880a800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f880c161fc0 0x7f880c0cc3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:38238/0 (socket says 172.21.15.90:38238) 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f880a800640 1 -- 172.21.15.90:0/2737226747 learned_addr learned my addr 172.21.15.90:0/2737226747 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f8809e00640 1 -- 172.21.15.90:0/2737226747 >> v1:172.21.15.102:6789/0 conn(0x7f880c070020 legacy=0x7f880c0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f8809e00640 1 -- 172.21.15.90:0/2737226747 >> v1:172.21.15.90:6789/0 conn(0x7f880c161fc0 legacy=0x7f880c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f8809e00640 1 -- 172.21.15.90:0/2737226747 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c0d0240 con 0x7f880c1619c0 2024-03-31T15:56:42.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.252+0000 7f8808a00640 1 -- 172.21.15.90:0/2737226747 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417775855 0 0) 0x7f880c0705c0 con 0x7f880c161fc0 2024-03-31T15:56:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f8809e00640 1 --2- 172.21.15.90:0/2737226747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c0d2850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0097c0 tx=0x7f87fc01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ed495437208bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f8808a00640 1 -- 172.21.15.90:0/2737226747 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0b0070 con 0x7f880c1619c0 2024-03-31T15:56:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f8808a00640 1 -- 172.21.15.90:0/2737226747 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc0ab600 con 0x7f880c1619c0 2024-03-31T15:56:42.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f8808a00640 1 -- 172.21.15.90:0/2737226747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc0ab9d0 con 0x7f880c1619c0 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 -- 172.21.15.90:0/2737226747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 msgr2=0x7f880c0d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 --2- 172.21.15.90:0/2737226747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c0d2850 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0097c0 tx=0x7f87fc01c380 comp rx=0 tx=0).stop 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 -- 172.21.15.90:0/2737226747 shutdown_connections 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 --2- 172.21.15.90:0/2737226747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c0d2850 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 -- 172.21.15.90:0/2737226747 >> 172.21.15.90:0/2737226747 conn(0x7f880c0d1d60 msgr2=0x7f880c0d2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 -- 172.21.15.90:0/2737226747 shutdown_connections 2024-03-31T15:56:42.250 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 -- 172.21.15.90:0/2737226747 wait complete. 2024-03-31T15:56:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.256+0000 7f880b200640 1 Processor -- start 2024-03-31T15:56:42.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- start start 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f880c070020 0x7f880c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f880c161fc0 0x7f880c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f880c16cb20 con 0x7f880c1619c0 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f880c162820 con 0x7f880c070020 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f880c16cca0 con 0x7f880c161fc0 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f880c070020 0x7f880c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59852/0 (socket says 172.21.15.90:59852) 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 -- 172.21.15.90:0/3289396224 learned_addr learned my addr 172.21.15.90:0/3289396224 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880a800640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f880c161fc0 0x7f880c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 -- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f880c161fc0 msgr2=0x7f880c06db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f880c161fc0 0x7f880c06db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 -- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f880c070020 msgr2=0x7f880c1570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f880c070020 0x7f880c1570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 -- 172.21.15.90:0/3289396224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c146630 con 0x7f880c1619c0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8809400640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c3c0 tx=0x7f87f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f8017070 con 0x7f880c1619c0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87f80052d0 con 0x7f880c1619c0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c157b90 con 0x7f880c1619c0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f8003840 con 0x7f880c1619c0 2024-03-31T15:56:42.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c158150 con 0x7f880c1619c0 2024-03-31T15:56:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.260+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f880c068d00 con 0x7f880c1619c0 2024-03-31T15:56:42.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.264+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87f801a070 con 0x7f880c1619c0 2024-03-31T15:56:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.264+0000 7f8802000640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 0x7f87ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.264+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f87f8085df0 con 0x7f880c1619c0 2024-03-31T15:56:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.264+0000 7f8809e00640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 0x7f87ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.264+0000 7f8809e00640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 0x7f87ec06a840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0ad000 tx=0x7f87fc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.268+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f87f8005480 con 0x7f880c1619c0 2024-03-31T15:56:42.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.424+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f880c06c3e0 con 0x7f880c1619c0 2024-03-31T15:56:42.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f8802000640 1 -- 172.21.15.90:0/3289396224 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7f87f8055bc0 con 0x7f880c1619c0 2024-03-31T15:56:42.420 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 msgr2=0x7f87ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 0x7f87ec06a840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f87fc0ad000 tx=0x7f87fc0a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 msgr2=0x7f880c06d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f87f800c3c0 tx=0x7f87f800c880 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 shutdown_connections 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f87ec068380 0x7f87ec06a840 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f880c161fc0 0x7f880c06db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f880c1619c0 0x7f880c06d620 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 --2- 172.21.15.90:0/3289396224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f880c070020 0x7f880c1570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 >> 172.21.15.90:0/3289396224 conn(0x7f880c0d1d60 msgr2=0x7f880c160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 shutdown_connections 2024-03-31T15:56:42.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.428+0000 7f880b200640 1 -- 172.21.15.90:0/3289396224 wait complete. 2024-03-31T15:56:42.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 Processor -- start 2024-03-31T15:56:42.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 -- start start 2024-03-31T15:56:42.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8de40cd640 0x7f8de40af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de40cefc0 con 0x7f8de415c320 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8de40aed70 con 0x7f8de40cd640 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de9800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8de40cf2a0 con 0x7f8de40d2b20 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:56214/0 (socket says 172.21.15.90:56214) 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 -- 172.21.15.90:0/1549348161 learned_addr learned my addr 172.21.15.90:0/1549348161 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:42.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de3400640 1 --2- 172.21.15.90:0/1549348161 >> v2:172.21.15.102:3300/0 conn(0x7f8de40cd640 0x7f8de40af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 -- 172.21.15.90:0/1549348161 >> v2:172.21.15.102:3300/0 conn(0x7f8de40cd640 msgr2=0x7f8de40af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 --2- 172.21.15.90:0/1549348161 >> v2:172.21.15.102:3300/0 conn(0x7f8de40cd640 0x7f8de40af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 -- 172.21.15.90:0/1549348161 >> v1:172.21.15.102:6789/0 conn(0x7f8de415c320 legacy=0x7f8de40cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.592+0000 7f8de2000640 1 -- 172.21.15.90:0/1549348161 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de40cf580 con 0x7f8de40d2b20 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de2000640 1 --2- 172.21.15.90:0/1549348161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009870 tx=0x7f8dd001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c906b9d20d0b8b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de1600640 1 -- 172.21.15.90:0/1549348161 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00b1070 con 0x7f8de40d2b20 2024-03-31T15:56:42.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de1600640 1 -- 172.21.15.90:0/1549348161 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd00a8050 con 0x7f8de40d2b20 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de1600640 1 -- 172.21.15.90:0/1549348161 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00ac860 con 0x7f8de40d2b20 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 -- 172.21.15.90:0/1549348161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 msgr2=0x7f8de40cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 --2- 172.21.15.90:0/1549348161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009870 tx=0x7f8dd001c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 -- 172.21.15.90:0/1549348161 shutdown_connections 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 --2- 172.21.15.90:0/1549348161 >> v2:172.21.15.102:3300/0 conn(0x7f8de40cd640 0x7f8de40af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 --2- 172.21.15.90:0/1549348161 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de40d2b20 0x7f8de40cd100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 -- 172.21.15.90:0/1549348161 >> 172.21.15.90:0/1549348161 conn(0x7f8de40d1f20 msgr2=0x7f8de40d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 -- 172.21.15.90:0/1549348161 shutdown_connections 2024-03-31T15:56:42.592 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.596+0000 7f8de9800640 1 -- 172.21.15.90:0/1549348161 wait complete. 2024-03-31T15:56:42.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 Processor -- start 2024-03-31T15:56:42.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- start start 2024-03-31T15:56:42.593 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8de40cd640 0x7f8de415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de415c320 0x7f8de40b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8de40b0360 con 0x7f8de40d2b20 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8de40b01e0 con 0x7f8de40cd640 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8de40b04e0 con 0x7f8de415c320 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:59856/0 (socket says 172.21.15.90:59856) 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 -- 172.21.15.90:0/1302793737 learned_addr learned my addr 172.21.15.90:0/1302793737 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de3400640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de415c320 0x7f8de40b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.594 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2a00640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8de40cd640 0x7f8de415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.595 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 -- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de415c320 msgr2=0x7f8de40b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de415c320 0x7f8de40b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 -- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8de40cd640 msgr2=0x7f8de415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8de40cd640 0x7f8de415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 -- 172.21.15.90:0/1302793737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de40b6560 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de2000640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8dd001ab80 tx=0x7f8dd001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00b1040 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd00bd070 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00b7660 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de415c970 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.600+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de415cf30 con 0x7f8de40d2b20 2024-03-31T15:56:42.596 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de40685f0 con 0x7f8de40d2b20 2024-03-31T15:56:42.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8dd00b7800 con 0x7f8de40d2b20 2024-03-31T15:56:42.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8ddaa00640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 0x7f8dc406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f8dd0120e30 con 0x7f8de40d2b20 2024-03-31T15:56:42.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8de2a00640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 0x7f8dc406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.604+0000 7f8de2a00640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 0x7f8dc406a840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4001810 tx=0x7f8dd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.608+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8dd0016150 con 0x7f8de40d2b20 2024-03-31T15:56:42.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.764+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8de40cc120 con 0x7f8de40d2b20 2024-03-31T15:56:42.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8ddaa00640 1 -- 172.21.15.90:0/1302793737 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f8dd00f0af0 con 0x7f8de40d2b20 2024-03-31T15:56:42.760 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:42.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 msgr2=0x7f8dc406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 0x7f8dc406a840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4001810 tx=0x7f8dd4009040 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 msgr2=0x7f8de40b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8dd001ab80 tx=0x7f8dd001ad10 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 shutdown_connections 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8dc4068380 0x7f8dc406a840 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8de415c320 0x7f8de40b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8de40d2b20 0x7f8de40b23e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 --2- 172.21.15.90:0/1302793737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8de40cd640 0x7f8de415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 >> 172.21.15.90:0/1302793737 conn(0x7f8de40d1f20 msgr2=0x7f8de415b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 shutdown_connections 2024-03-31T15:56:42.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.768+0000 7f8de9800640 1 -- 172.21.15.90:0/1302793737 wait complete. 2024-03-31T15:56:42.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:42.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 Processor -- start 2024-03-31T15:56:42.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 -- start start 2024-03-31T15:56:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 0x7f2c000b2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c000d2df0 con 0x7f2c0015e320 2024-03-31T15:56:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0015e700 con 0x7f2c000d2940 2024-03-31T15:56:42.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.936+0000 7f2c07600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2c000d1400 con 0x7f2c001624a0 2024-03-31T15:56:42.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2c05600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 0x7f2c000b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50408/0 (socket says 172.21.15.90:50408) 2024-03-31T15:56:42.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 -- 172.21.15.90:0/2446333559 learned_addr learned my addr 172.21.15.90:0/2446333559 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2c04c00640 1 --1- 172.21.15.90:0/2446333559 >> v1:172.21.15.102:6789/0 conn(0x7f2c0015e320 0x7f2c00161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:54278/0 (socket says 172.21.15.90:54278) 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 -- 172.21.15.90:0/2446333559 >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 msgr2=0x7f2c000b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 --2- 172.21.15.90:0/2446333559 >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 0x7f2c000b2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 -- 172.21.15.90:0/2446333559 >> v1:172.21.15.102:6789/0 conn(0x7f2c0015e320 legacy=0x7f2c00161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 -- 172.21.15.90:0/2446333559 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c000b1a20 con 0x7f2c001624a0 2024-03-31T15:56:42.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2c05600640 1 --2- 172.21.15.90:0/2446333559 >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 0x7f2c000b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.940+0000 7f2bffe00640 1 --2- 172.21.15.90:0/2446333559 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009870 tx=0x7f2bec01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d23f0b3d0c838a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2bff400640 1 -- 172.21.15.90:0/2446333559 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0a9640 con 0x7f2c001624a0 2024-03-31T15:56:42.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2bff400640 1 -- 172.21.15.90:0/2446333559 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bec0a97e0 con 0x7f2c001624a0 2024-03-31T15:56:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2bff400640 1 -- 172.21.15.90:0/2446333559 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0a9bb0 con 0x7f2c001624a0 2024-03-31T15:56:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- 172.21.15.90:0/2446333559 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 msgr2=0x7f2c000d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- 172.21.15.90:0/2446333559 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bec009870 tx=0x7f2bec01c560 comp rx=0 tx=0).stop 2024-03-31T15:56:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- 172.21.15.90:0/2446333559 shutdown_connections 2024-03-31T15:56:42.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- 172.21.15.90:0/2446333559 >> v2:172.21.15.102:3300/0 conn(0x7f2c000d2940 0x7f2c000b2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- 172.21.15.90:0/2446333559 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c001624a0 0x7f2c000d0d20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- 172.21.15.90:0/2446333559 >> 172.21.15.90:0/2446333559 conn(0x7f2c0008bfd0 msgr2=0x7f2c000d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- 172.21.15.90:0/2446333559 shutdown_connections 2024-03-31T15:56:42.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- 172.21.15.90:0/2446333559 wait complete. 2024-03-31T15:56:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 Processor -- start 2024-03-31T15:56:42.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- start start 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c000d2940 0x7f2c00156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0015e320 0x7f2c000ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c000b35e0 con 0x7f2c001624a0 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2c000b3460 con 0x7f2c0015e320 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c07600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f2c000b3760 con 0x7f2c000d2940 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c05600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c05600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41484/0 (socket says 172.21.15.90:41484) 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c05600640 1 -- 172.21.15.90:0/1097547535 learned_addr learned my addr 172.21.15.90:0/1097547535 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2bffe00640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0015e320 0x7f2c000ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.944+0000 7f2c04c00640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c000d2940 0x7f2c00156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 -- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c000d2940 msgr2=0x7f2c00156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c000d2940 0x7f2c00156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 -- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0015e320 msgr2=0x7f2c000ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0015e320 0x7f2c000ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 -- 172.21.15.90:0/1097547535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c000c92e0 con 0x7f2c001624a0 2024-03-31T15:56:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c05600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800efc0 tx=0x7f2bf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8018030 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf800cdf0 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8003bc0 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c000c95c0 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c000c9b80 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c00068660 con 0x7f2c001624a0 2024-03-31T15:56:42.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2bf801b070 con 0x7f2c001624a0 2024-03-31T15:56:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.948+0000 7f2bfcc00640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 0x7f2be006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:42.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.952+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f2bf80875c0 con 0x7f2c001624a0 2024-03-31T15:56:42.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.952+0000 7f2c04c00640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 0x7f2be006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:42.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.952+0000 7f2c04c00640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 0x7f2be006a840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4000f70 tx=0x7f2bf4007cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:42.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:42.956+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f2bf8010040 con 0x7f2c001624a0 2024-03-31T15:56:43.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2c000d1800 con 0x7f2c001624a0 2024-03-31T15:56:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2bfcc00640 1 -- 172.21.15.90:0/1097547535 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7f2bf80578d0 con 0x7f2c001624a0 2024-03-31T15:56:43.129 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 msgr2=0x7f2be006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 0x7f2be006a840 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4000f70 tx=0x7f2bf4007cd0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 msgr2=0x7f2c000ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800efc0 tx=0x7f2bf800c490 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 shutdown_connections 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2be0068380 0x7f2be006a840 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c001624a0 0x7f2c000ce890 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c0015e320 0x7f2c000ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 --2- 172.21.15.90:0/1097547535 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2c000d2940 0x7f2c00156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 >> 172.21.15.90:0/1097547535 conn(0x7f2c0008bfd0 msgr2=0x7f2c0006cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 shutdown_connections 2024-03-31T15:56:43.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.136+0000 7f2c07600640 1 -- 172.21.15.90:0/1097547535 wait complete. 2024-03-31T15:56:43.149 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:56:43.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:43.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[41631]: fuse finished with error 0 and tester_r 0 2024-03-31T15:56:43.169 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:43.249 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:43.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.250 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:56:43.259 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:56:43.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 Processor -- start 2024-03-31T15:56:43.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 -- start start 2024-03-31T15:56:43.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6801449a0 0x7fd68015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd68015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6800b47e0 con 0x7fd6800b3d90 2024-03-31T15:56:43.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd6800b4ac0 con 0x7fd6801449a0 2024-03-31T15:56:43.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd686400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd68015b7d0 con 0x7fd68006bf30 2024-03-31T15:56:43.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd6800b3d90 0x7fd68006b820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60656/0 (socket says 172.21.15.90:60656) 2024-03-31T15:56:43.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 -- 172.21.15.90:0/4247764666 learned_addr learned my addr 172.21.15.90:0/4247764666 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:43.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67f400640 1 --2- 172.21.15.90:0/4247764666 >> v2:172.21.15.102:3300/0 conn(0x7fd6801449a0 0x7fd68015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67fe00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd68015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3198323898 0 0) 0x7fd6800b47e0 con 0x7fd6800b3d90 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd664005740 con 0x7fd6800b3d90 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3145656227 0 0) 0x7fd664005740 con 0x7fd6800b3d90 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 msgr2=0x7fd68015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd68015ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 >> v2:172.21.15.102:3300/0 conn(0x7fd6801449a0 msgr2=0x7fd68015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 --2- 172.21.15.90:0/4247764666 >> v2:172.21.15.102:3300/0 conn(0x7fd6801449a0 0x7fd68015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68015bab0 con 0x7fd6800b3d90 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fd668001680 con 0x7fd6800b3d90 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 >> v1:172.21.15.90:6789/0 conn(0x7fd6800b3d90 legacy=0x7fd68006b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664005bd0 0x7fd664005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd664006a20 0x7fd664006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd6640073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800b4ac0 con 0x7fd664006a20 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd68015b7d0 con 0x7fd664005bd0 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd66400dbe0 con 0x7fd68006bf30 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd664006a20 0x7fd664006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67f400640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664005bd0 0x7fd664005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd67fe00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd6640073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 -- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 msgr2=0x7fd6640073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68006bf30 0x7fd6640073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 -- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664005bd0 msgr2=0x7fd664005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664005bd0 0x7fd664005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.496+0000 7fd677e00640 1 -- 172.21.15.90:0/4247764666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd664005740 con 0x7fd664006a20 2024-03-31T15:56:43.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd677e00640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd664006a20 0x7fd664006ea0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c160 tx=0x7fd66801f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18960c0de3c2236 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:43.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668007c90 con 0x7fd664006a20 2024-03-31T15:56:43.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6680ac020 con 0x7fd664006a20 2024-03-31T15:56:43.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd67ea00640 1 -- 172.21.15.90:0/4247764666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6680058e0 con 0x7fd664006a20 2024-03-31T15:56:43.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 -- 172.21.15.90:0/4247764666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd664006a20 msgr2=0x7fd664006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 --2- 172.21.15.90:0/4247764666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd664006a20 0x7fd664006ea0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd66800c160 tx=0x7fd66801f330 comp rx=0 tx=0).stop 2024-03-31T15:56:43.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd67fe00640 1 -- 172.21.15.90:0/4247764666 reap_dead start 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 -- 172.21.15.90:0/4247764666 shutdown_connections 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 -- 172.21.15.90:0/4247764666 >> 172.21.15.90:0/4247764666 conn(0x7fd6800b1c80 msgr2=0x7fd6800b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 -- 172.21.15.90:0/4247764666 shutdown_connections 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.500+0000 7fd686400640 1 -- 172.21.15.90:0/4247764666 wait complete. 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 Processor -- start 2024-03-31T15:56:43.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- start start 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 0x7fd6800cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68015aa60 0x7fd6800ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800b4220 con 0x7fd680153a60 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd68015b450 con 0x7fd664006a20 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd6800b43a0 con 0x7fd68015aa60 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41508/0 (socket says 172.21.15.90:41508) 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 -- 172.21.15.90:0/36002846 learned_addr learned my addr 172.21.15.90:0/36002846 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67f400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 0x7fd6800cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd677e00640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68015aa60 0x7fd6800ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 -- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68015aa60 msgr2=0x7fd6800ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68015aa60 0x7fd6800ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 -- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 msgr2=0x7fd6800cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 0x7fd6800cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 -- 172.21.15.90:0/36002846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6800c9480 con 0x7fd680153a60 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67f400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 0x7fd6800cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd67fe00640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd680071af0 tx=0x7fd6780115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678011e40 con 0x7fd680153a60 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67801b070 con 0x7fd680153a60 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6780039a0 con 0x7fd680153a60 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6800c9760 con 0x7fd680153a60 2024-03-31T15:56:43.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.504+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6800c9d20 con 0x7fd680153a60 2024-03-31T15:56:43.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6800686d0 con 0x7fd680153a60 2024-03-31T15:56:43.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd678003b40 con 0x7fd680153a60 2024-03-31T15:56:43.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd677400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 0x7fd65c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:43.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fd6780872e0 con 0x7fd680153a60 2024-03-31T15:56:43.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd67f400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 0x7fd65c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:43.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.508+0000 7fd67f400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 0x7fd65c06a840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd670009be0 tx=0x7fd670008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:43.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.512+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd678025020 con 0x7fd680153a60 2024-03-31T15:56:43.698 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.704+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd68015af70 con 0x7fd680153a60 2024-03-31T15:56:43.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.704+0000 7fd677400640 1 -- 172.21.15.90:0/36002846 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7fd678057700 con 0x7fd680153a60 2024-03-31T15:56:43.699 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 msgr2=0x7fd65c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 0x7fd65c06a840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd670009be0 tx=0x7fd670008040 comp rx=0 tx=0).stop 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 msgr2=0x7fd68013e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd680071af0 tx=0x7fd6780115a0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 shutdown_connections 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd65c068380 0x7fd65c06a840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd680153a60 0x7fd68013e2e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd68015aa60 0x7fd6800ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.701 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 --2- 172.21.15.90:0/36002846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd664006a20 0x7fd6800cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:43.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 >> 172.21.15.90:0/36002846 conn(0x7fd6800b1c80 msgr2=0x7fd68015e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:43.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 shutdown_connections 2024-03-31T15:56:43.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43.708+0000 7fd686400640 1 -- 172.21.15.90:0/36002846 wait complete. 2024-03-31T15:56:43.719 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:56:43.719 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:56:43.720 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:43.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.720 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42320sec preferred_lft 42320sec 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:56:43.731 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:56:43.732 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:43.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.733 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:56:43.792 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:56:43.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.795 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:56:43.846 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:56:43.846 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:56:43.846 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:43.847 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:56:43.847 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:56:43.847 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:56:43.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:56:43.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:56:43.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:56:43.848 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:56:43.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:56:43.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:43.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.980 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:56:43.989 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:56:43.989 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:56:43.989 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:43.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:43.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:56:44.054 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:56:44.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:44.054 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:44.107 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:44.107 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:44.108 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:44.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:44.219 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:44.272 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:44.273 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:44.274 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:44.275 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:44.275 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:44.275 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:44.275 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:44.275 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:44.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:44.276 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:44.327 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:56:44.327 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:56:44.370 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:44.390 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:44.442 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:44.443 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:44.445 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:44.445 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:44.445 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:44.445 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:44.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:44.445 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:44.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:56:44.468+0000 7f42731db580 -1 init, newargv = 0x557ab68b32c0 newargc=15 2024-03-31T15:56:44.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42032]: starting ceph client 2024-03-31T15:56:44.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42032]: starting fuse 2024-03-31T15:56:44.471 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:56:44.472 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:56:44.472 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:56:44.473 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:56:44.474 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:56:44.595 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.42032.asok 2024-03-31T15:56:44.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:44.606 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42032.asok 2024-03-31T15:56:44.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:44.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42032.asok status 2024-03-31T15:56:44.775 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:56:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "42032", 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6468, 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:56:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6468 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1173267724 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1173267724 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6468 192.168.0.1:0/1173267724", 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/1173267724", 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 27, 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 55, 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:56:44.778 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:56:44.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:44.791 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:56:44.791 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:44.800 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:56:44.800 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:44.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:44.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> import errno 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> if False: 2024-03-31T15:56:44.866 DEBUG:teuthology.orchestra.run.smithi090:> ret = os.read(fd, 1024) 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> else: 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> os.write(fd, b'"'"'content'"'"') 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> except OSError as e: 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> if e.errno != errno.EPERM: 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> raise 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> else: 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("client does not check permission of data pool") 2024-03-31T15:56:44.867 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:56:44.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:44.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-31T15:56:45.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 Processor -- start 2024-03-31T15:56:45.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 -- start start 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f583015c320 0x7f58300cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300d2b20 0x7f58300ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f58300cd470 0x7f58300cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.176+0000 7f5836e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f58300aed70 con 0x7f583015c320 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f5836e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f58300cd900 con 0x7f58300cd470 2024-03-31T15:56:45.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f5836e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f58300cefc0 con 0x7f58300d2b20 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f583015c320 0x7f58300cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f583015c320 0x7f58300cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41520/0 (socket says 172.21.15.90:41520) 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 -- 172.21.15.90:0/920634139 learned_addr learned my addr 172.21.15.90:0/920634139 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582f400640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300d2b20 0x7f58300ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f5834e00640 1 --2- 172.21.15.90:0/920634139 >> v2:172.21.15.102:3300/0 conn(0x7f58300cd470 0x7f58300cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 -- 172.21.15.90:0/920634139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300d2b20 msgr2=0x7f58300ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300d2b20 0x7f58300ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 -- 172.21.15.90:0/920634139 >> v2:172.21.15.102:3300/0 conn(0x7f58300cd470 msgr2=0x7f58300cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 --2- 172.21.15.90:0/920634139 >> v2:172.21.15.102:3300/0 conn(0x7f58300cd470 0x7f58300cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 -- 172.21.15.90:0/920634139 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58300cf2a0 con 0x7f583015c320 2024-03-31T15:56:45.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582fe00640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f58300cc9f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f583015fdd0 tx=0x7f582001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54545d7b8c3a954c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582ea00640 1 -- 172.21.15.90:0/920634139 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58200b0070 con 0x7f583015c320 2024-03-31T15:56:45.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582ea00640 1 -- 172.21.15.90:0/920634139 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58200ab600 con 0x7f583015c320 2024-03-31T15:56:45.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.180+0000 7f582ea00640 1 -- 172.21.15.90:0/920634139 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58200ab9d0 con 0x7f583015c320 2024-03-31T15:56:45.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- 172.21.15.90:0/920634139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 msgr2=0x7f58300cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f58300cc9f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f583015fdd0 tx=0x7f582001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:45.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- 172.21.15.90:0/920634139 shutdown_connections 2024-03-31T15:56:45.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- 172.21.15.90:0/920634139 >> v2:172.21.15.102:3300/0 conn(0x7f58300cd470 0x7f58300cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300d2b20 0x7f58300ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- 172.21.15.90:0/920634139 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f58300cc9f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- 172.21.15.90:0/920634139 >> 172.21.15.90:0/920634139 conn(0x7f58300d1f20 msgr2=0x7f58300d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:45.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- 172.21.15.90:0/920634139 shutdown_connections 2024-03-31T15:56:45.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- 172.21.15.90:0/920634139 wait complete. 2024-03-31T15:56:45.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 Processor -- start 2024-03-31T15:56:45.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- start start 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300cd470 0x7f5830154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 0x7f5830157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58300d0e20 con 0x7f583015c320 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58300d0ca0 con 0x7f58300d2b20 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5836e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f58300af200 con 0x7f58300cd470 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5834e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5834e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41524/0 (socket says 172.21.15.90:41524) 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f5834e00640 1 -- 172.21.15.90:0/3663086500 learned_addr learned my addr 172.21.15.90:0/3663086500 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f582f400640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 0x7f5830157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.184+0000 7f582fe00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300cd470 0x7f5830154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 -- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300cd470 msgr2=0x7f5830154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300cd470 0x7f5830154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 -- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 msgr2=0x7f5830157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 0x7f5830157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 -- 172.21.15.90:0/3663086500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583015db70 con 0x7f583015c320 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f582f400640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 0x7f5830157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5834e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5828007e40 tx=0x7f582800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828014620 con 0x7f583015c320 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5828019070 con 0x7f583015c320 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583015de50 con 0x7f583015c320 2024-03-31T15:56:45.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828014a20 con 0x7f583015c320 2024-03-31T15:56:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583015e410 con 0x7f583015c320 2024-03-31T15:56:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58300685f0 con 0x7f583015c320 2024-03-31T15:56:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.188+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5828014bc0 con 0x7f583015c320 2024-03-31T15:56:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.192+0000 7f5827e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 0x7f581006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.192+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f58280873f0 con 0x7f583015c320 2024-03-31T15:56:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.192+0000 7f582fe00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 0x7f581006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.192+0000 7f582fe00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 0x7f581006a870 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f58200ad000 tx=0x7f58200a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.196+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5828057820 con 0x7f583015c320 2024-03-31T15:56:45.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.408+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) -- 0x7f5830070430 con 0x7f583015c320 2024-03-31T15:56:45.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5827e00640 1 -- 172.21.15.90:0/3663086500 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v19) ==== 183+0+140 (secure 0 0 0) 0x7f5828057120 con 0x7f583015c320 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-31T15:56:45.405 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 msgr2=0x7f581006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 0x7f581006a870 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f58200ad000 tx=0x7f58200a4f90 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 msgr2=0x7f5830157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5828007e40 tx=0x7f582800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 shutdown_connections 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f58100683b0 0x7f581006a870 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f583015c320 0x7f5830157580 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58300d2b20 0x7f5830157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 --2- 172.21.15.90:0/3663086500 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f58300cd470 0x7f5830154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 >> 172.21.15.90:0/3663086500 conn(0x7f58300d1f20 msgr2=0x7f58300cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:45.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 shutdown_connections 2024-03-31T15:56:45.408 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.412+0000 7f5836e00640 1 -- 172.21.15.90:0/3663086500 wait complete. 2024-03-31T15:56:45.427 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:45.427 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:45.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:45.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 Processor -- start 2024-03-31T15:56:45.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 -- start start 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f628c15c120 0x7f628c15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0cca30 0x7f628c0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628c0d2b10 con 0x7f628c0d2680 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f628c160bf0 con 0x7f628c15c120 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6292c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f628c160350 con 0x7f628c0cca30 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f628c15c120 0x7f628c15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6283e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f628c0d2680 0x7f628c0b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60660/0 (socket says 172.21.15.90:60660) 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6283e00640 1 -- 172.21.15.90:0/843079758 learned_addr learned my addr 172.21.15.90:0/843079758 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:45.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f6290c00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0cca30 0x7f628c0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628be00640 1 -- 172.21.15.90:0/843079758 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0cca30 msgr2=0x7f628c0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:45.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628be00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0cca30 0x7f628c0ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628be00640 1 -- 172.21.15.90:0/843079758 >> v1:172.21.15.90:6789/0 conn(0x7f628c0d2680 legacy=0x7f628c0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628be00640 1 -- 172.21.15.90:0/843079758 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c160630 con 0x7f628c15c120 2024-03-31T15:56:45.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.620+0000 7f628b400640 1 -- 172.21.15.90:0/843079758 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169139531 0 0) 0x7f628c0d2b10 con 0x7f628c0d2680 2024-03-31T15:56:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f628be00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15fd00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f627c0097c0 tx=0x7f627c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604da62412781d6f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f628b400640 1 -- 172.21.15.90:0/843079758 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0b0070 con 0x7f628c15c120 2024-03-31T15:56:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f628b400640 1 -- 172.21.15.90:0/843079758 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f627c0ab600 con 0x7f628c15c120 2024-03-31T15:56:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f628b400640 1 -- 172.21.15.90:0/843079758 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0ab9d0 con 0x7f628c15c120 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 -- 172.21.15.90:0/843079758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 msgr2=0x7f628c15fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15fd00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f627c0097c0 tx=0x7f627c01c410 comp rx=0 tx=0).stop 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 -- 172.21.15.90:0/843079758 shutdown_connections 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0cca30 0x7f628c0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 --2- 172.21.15.90:0/843079758 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15fd00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 -- 172.21.15.90:0/843079758 >> 172.21.15.90:0/843079758 conn(0x7f628c09aa40 msgr2=0x7f628c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 -- 172.21.15.90:0/843079758 shutdown_connections 2024-03-31T15:56:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 -- 172.21.15.90:0/843079758 wait complete. 2024-03-31T15:56:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.624+0000 7f6292c00640 1 Processor -- start 2024-03-31T15:56:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- start start 2024-03-31T15:56:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f628c0cca30 0x7f628c070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0d2680 0x7f628c070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f628c0b2f70 con 0x7f628c15c120 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f628c0b2df0 con 0x7f628c0cca30 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f628c169040 con 0x7f628c0d2680 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41544/0 (socket says 172.21.15.90:41544) 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 -- 172.21.15.90:0/1280483545 learned_addr learned my addr 172.21.15.90:0/1280483545 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6283e00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0d2680 0x7f628c070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f628be00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f628c0cca30 0x7f628c070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 -- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0d2680 msgr2=0x7f628c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0d2680 0x7f628c070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 -- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f628c0cca30 msgr2=0x7f628c070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f628c0cca30 0x7f628c070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 -- 172.21.15.90:0/1280483545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c0c8880 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6290c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6284007e60 tx=0x7f628400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284014420 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6284019070 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c0c85a0 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284014820 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c15cb60 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.628+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c0686d0 con 0x7f628c15c120 2024-03-31T15:56:45.625 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.632+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62840149c0 con 0x7f628c15c120 2024-03-31T15:56:45.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.632+0000 7f6288c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 0x7f626806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.632+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f6284087480 con 0x7f628c15c120 2024-03-31T15:56:45.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.632+0000 7f628be00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 0x7f626806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.632+0000 7f628be00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 0x7f626806a870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f627c0ad000 tx=0x7f627c0a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.636+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6284057780 con 0x7f628c15c120 2024-03-31T15:56:45.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.792+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f628c0cc350 con 0x7f628c15c120 2024-03-31T15:56:45.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6288c00640 1 -- 172.21.15.90:0/1280483545 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f6284057080 con 0x7f628c15c120 2024-03-31T15:56:45.788 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:45.788 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 msgr2=0x7f626806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 0x7f626806a870 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f627c0ad000 tx=0x7f627c0a4f90 comp rx=0 tx=0).stop 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 msgr2=0x7f628c15edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6284007e60 tx=0x7f628400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 shutdown_connections 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f62680683b0 0x7f626806a870 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f628c15c120 0x7f628c15edd0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f628c0d2680 0x7f628c070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 --2- 172.21.15.90:0/1280483545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f628c0cca30 0x7f628c070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 >> 172.21.15.90:0/1280483545 conn(0x7f628c09aa40 msgr2=0x7f628c0cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:45.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 shutdown_connections 2024-03-31T15:56:45.791 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.796+0000 7f6292c00640 1 -- 172.21.15.90:0/1280483545 wait complete. 2024-03-31T15:56:45.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:45.809 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:45.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.952+0000 7f979e600640 1 Processor -- start 2024-03-31T15:56:45.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.952+0000 7f979e600640 1 -- start start 2024-03-31T15:56:45.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f979e600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f979815c320 0x7f97980cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f979e600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97980cefc0 con 0x7f97980d2b20 2024-03-31T15:56:45.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f979e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97980cf2a0 con 0x7f97980cd640 2024-03-31T15:56:45.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f979e600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f97980aed70 con 0x7f979815c320 2024-03-31T15:56:45.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f979815c320 0x7f97980cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f979815c320 0x7f97980cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41560/0 (socket says 172.21.15.90:41560) 2024-03-31T15:56:45.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 -- 172.21.15.90:0/4162902605 learned_addr learned my addr 172.21.15.90:0/4162902605 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:45.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 -- 172.21.15.90:0/4162902605 >> v1:172.21.15.102:6789/0 conn(0x7f97980cd640 legacy=0x7f97980af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 -- 172.21.15.90:0/4162902605 >> v1:172.21.15.90:6789/0 conn(0x7f97980d2b20 legacy=0x7f97980ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.956+0000 7f9797400640 1 -- 172.21.15.90:0/4162902605 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97980cf580 con 0x7f979815c320 2024-03-31T15:56:45.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f9797400640 1 --2- 172.21.15.90:0/4162902605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f979815c320 0x7f97980cc9f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9788009870 tx=0x7f978801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d606f9136fb9b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f9796a00640 1 -- 172.21.15.90:0/4162902605 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880b1070 con 0x7f979815c320 2024-03-31T15:56:45.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f9796a00640 1 -- 172.21.15.90:0/4162902605 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97880a8050 con 0x7f979815c320 2024-03-31T15:56:45.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f9796a00640 1 -- 172.21.15.90:0/4162902605 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97880ac860 con 0x7f979815c320 2024-03-31T15:56:45.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- 172.21.15.90:0/4162902605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f979815c320 msgr2=0x7f97980cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 --2- 172.21.15.90:0/4162902605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f979815c320 0x7f97980cc9f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9788009870 tx=0x7f978801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- 172.21.15.90:0/4162902605 shutdown_connections 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 --2- 172.21.15.90:0/4162902605 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f979815c320 0x7f97980cc9f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- 172.21.15.90:0/4162902605 >> 172.21.15.90:0/4162902605 conn(0x7f97980d1f20 msgr2=0x7f97980d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- 172.21.15.90:0/4162902605 shutdown_connections 2024-03-31T15:56:45.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- 172.21.15.90:0/4162902605 wait complete. 2024-03-31T15:56:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 Processor -- start 2024-03-31T15:56:45.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.960+0000 7f979e600640 1 -- start start 2024-03-31T15:56:45.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97980cd640 0x7f979815f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979815c320 0x7f97981fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f97980b04f0 con 0x7f97980d2b20 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97980b0370 con 0x7f979815c320 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9798169ed0 con 0x7f97980cd640 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41574/0 (socket says 172.21.15.90:41574) 2024-03-31T15:56:45.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 -- 172.21.15.90:0/2207361906 learned_addr learned my addr 172.21.15.90:0/2207361906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f9797400640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97980cd640 0x7f979815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f9797e00640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979815c320 0x7f97981fafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 -- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97980cd640 msgr2=0x7f979815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97980cd640 0x7f979815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 -- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979815c320 msgr2=0x7f97981fafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:45.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979815c320 0x7f97981fafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 -- 172.21.15.90:0/2207361906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979815db20 con 0x7f97980d2b20 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978fe00640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f978000c3f0 tx=0x7f978000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780017070 con 0x7f97980d2b20 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97800052d0 con 0x7f97980d2b20 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97981562b0 con 0x7f97980d2b20 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780003840 con 0x7f97980d2b20 2024-03-31T15:56:45.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9798156550 con 0x7f97980d2b20 2024-03-31T15:56:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.964+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97980685f0 con 0x7f97980d2b20 2024-03-31T15:56:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.968+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f978001a070 con 0x7f97980d2b20 2024-03-31T15:56:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.968+0000 7f978f400640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 0x7f977406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.968+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f9780085e30 con 0x7f97980d2b20 2024-03-31T15:56:45.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.968+0000 7f9797400640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 0x7f977406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:45.963 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.968+0000 7f9797400640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 0x7f977406a840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f97880ae000 tx=0x7f97880a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:45.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:45.972+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9780005480 con 0x7f97980d2b20 2024-03-31T15:56:46.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f979806c700 con 0x7f97980d2b20 2024-03-31T15:56:46.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f978f400640 1 -- 172.21.15.90:0/2207361906 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f9780055c10 con 0x7f97980d2b20 2024-03-31T15:56:46.125 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:46.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 msgr2=0x7f977406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 0x7f977406a840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f97880ae000 tx=0x7f97880a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 msgr2=0x7f97981faaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f978000c3f0 tx=0x7f978000c8b0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 shutdown_connections 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9774068380 0x7f977406a840 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979815c320 0x7f97981fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97980d2b20 0x7f97981faaa0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 --2- 172.21.15.90:0/2207361906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f97980cd640 0x7f979815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 >> 172.21.15.90:0/2207361906 conn(0x7f97980d1f20 msgr2=0x7f97980709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 shutdown_connections 2024-03-31T15:56:46.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.132+0000 7f979e600640 1 -- 172.21.15.90:0/2207361906 wait complete. 2024-03-31T15:56:46.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:46.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 Processor -- start 2024-03-31T15:56:46.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 -- start start 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa7400ace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7400ad3d0 0x7fa7400aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa74006e9c0 con 0x7fa7400af040 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa74006de40 con 0x7fa7400ad3d0 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.308+0000 7fa745e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa74006e120 con 0x7fa74015c290 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7400ad3d0 0x7fa7400aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa7400ad3d0 0x7fa7400aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41582/0 (socket says 172.21.15.90:41582) 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 -- 172.21.15.90:0/2059932512 learned_addr learned my addr 172.21.15.90:0/2059932512 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa73ea00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa7400ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 -- 172.21.15.90:0/2059932512 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 msgr2=0x7fa7400ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:46.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa7400ace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 -- 172.21.15.90:0/2059932512 >> v1:172.21.15.102:6789/0 conn(0x7fa7400af040 legacy=0x7fa74015fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 -- 172.21.15.90:0/2059932512 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74006e400 con 0x7fa7400ad3d0 2024-03-31T15:56:46.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa737e00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa7400aeb00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa728009870 tx=0x7fa72801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f80e1f1ef0a85e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa73e000640 1 -- 172.21.15.90:0/2059932512 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7280b2070 con 0x7fa7400ad3d0 2024-03-31T15:56:46.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa73e000640 1 -- 172.21.15.90:0/2059932512 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7280a9070 con 0x7fa7400ad3d0 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.312+0000 7fa73e000640 1 -- 172.21.15.90:0/2059932512 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7280ad880 con 0x7fa7400ad3d0 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- 172.21.15.90:0/2059932512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 msgr2=0x7fa7400aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa7400aeb00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa728009870 tx=0x7fa72801c560 comp rx=0 tx=0).stop 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- 172.21.15.90:0/2059932512 shutdown_connections 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa7400aeb00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 --2- 172.21.15.90:0/2059932512 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa7400ace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- 172.21.15.90:0/2059932512 >> 172.21.15.90:0/2059932512 conn(0x7fa7400b18b0 msgr2=0x7fa7400b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- 172.21.15.90:0/2059932512 shutdown_connections 2024-03-31T15:56:46.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- 172.21.15.90:0/2059932512 wait complete. 2024-03-31T15:56:46.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 Processor -- start 2024-03-31T15:56:46.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.316+0000 7fa745e00640 1 -- start start 2024-03-31T15:56:46.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7400af040 0x7fa7400cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa740154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa740169180 con 0x7fa7400ad3d0 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa740169000 con 0x7fa7400af040 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa740169300 con 0x7fa74015c290 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41588/0 (socket says 172.21.15.90:41588) 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 -- 172.21.15.90:0/2309507880 learned_addr learned my addr 172.21.15.90:0/2309507880 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73f400640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa740154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa737e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7400af040 0x7fa7400cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 -- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 msgr2=0x7fa740154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa740154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 -- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7400af040 msgr2=0x7fa7400cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7400af040 0x7fa7400cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 -- 172.21.15.90:0/2309507880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7400ce3f0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73f400640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa740154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa73ea00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa73000d620 tx=0x7fa73000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000b620 con 0x7fa7400ad3d0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa730019070 con 0x7fa7400ad3d0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74015a1d0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000ba20 con 0x7fa7400ad3d0 2024-03-31T15:56:46.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa74014f9d0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.320+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa740075880 con 0x7fa7400ad3d0 2024-03-31T15:56:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.324+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa73000bbc0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.324+0000 7fa737400640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 0x7fa71c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.324+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fa730086fb0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.324+0000 7fa737e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 0x7fa71c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.324+0000 7fa737e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 0x7fa71c06a870 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7280af000 tx=0x7fa7280a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.328+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa730057340 con 0x7fa7400ad3d0 2024-03-31T15:56:46.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.592+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7400685f0 con 0x7fa7400ad3d0 2024-03-31T15:56:46.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.592+0000 7fa737400640 1 -- 172.21.15.90:0/2309507880 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7fa730056c40 con 0x7fa7400ad3d0 2024-03-31T15:56:46.587 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 msgr2=0x7fa71c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 0x7fa71c06a870 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa7280af000 tx=0x7fa7280a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 msgr2=0x7fa740159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa73000d620 tx=0x7fa73000dae0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 shutdown_connections 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa71c0683b0 0x7fa71c06a870 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.589 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa74015c290 0x7fa740154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7400af040 0x7fa7400cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 --2- 172.21.15.90:0/2309507880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7400ad3d0 0x7fa740159bb0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 >> 172.21.15.90:0/2309507880 conn(0x7fa7400b18b0 msgr2=0x7fa740074a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:46.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 shutdown_connections 2024-03-31T15:56:46.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.596+0000 7fa745e00640 1 -- 172.21.15.90:0/2309507880 wait complete. 2024-03-31T15:56:46.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 Processor -- start 2024-03-31T15:56:46.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 -- start start 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804166ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280406e9c0 con 0x7f28040aeb00 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280406de40 con 0x7f280415c290 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f280a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f280406e120 con 0x7f2804072760 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f27fbe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f28040aeb00 0x7f28040aef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60674/0 (socket says 172.21.15.90:60674) 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f27fbe00640 1 -- 172.21.15.90:0/763506013 learned_addr learned my addr 172.21.15.90:0/763506013 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:46.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2803e00640 1 --2- 172.21.15.90:0/763506013 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804166ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195897095 0 0) 0x7f280406e9c0 con 0x7f28040aeb00 2024-03-31T15:56:46.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27e8003530 con 0x7f28040aeb00 2024-03-31T15:56:46.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2719217317 0 0) 0x7f280406de40 con 0x7f280415c290 2024-03-31T15:56:46.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f280406e9c0 con 0x7f280415c290 2024-03-31T15:56:46.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2803e00640 1 -- 172.21.15.90:0/763506013 >> v1:172.21.15.102:6789/0 conn(0x7f280415c290 legacy=0x7f28040ace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1407010220 0 0) 0x7f27e8003530 con 0x7f28040aeb00 2024-03-31T15:56:46.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2803e00640 1 -- 172.21.15.90:0/763506013 >> v1:172.21.15.90:6789/0 conn(0x7f28040aeb00 legacy=0x7f28040aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.756+0000 7f2803e00640 1 -- 172.21.15.90:0/763506013 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280406e400 con 0x7f2804072760 2024-03-31T15:56:46.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f2803e00640 1 --2- 172.21.15.90:0/763506013 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804166ff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009dc0 tx=0x7f27fc01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb308e1874e0290 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec001680 con 0x7f2804072760 2024-03-31T15:56:46.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc0a6500 con 0x7f2804072760 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f2802a00640 1 -- 172.21.15.90:0/763506013 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0a68d0 con 0x7f2804072760 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 -- 172.21.15.90:0/763506013 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 msgr2=0x7f2804166ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 --2- 172.21.15.90:0/763506013 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804166ff0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27fc009dc0 tx=0x7f27fc01c8b0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 -- 172.21.15.90:0/763506013 shutdown_connections 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 --2- 172.21.15.90:0/763506013 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804166ff0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 -- 172.21.15.90:0/763506013 >> 172.21.15.90:0/763506013 conn(0x7f28040b18b0 msgr2=0x7f28040b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 -- 172.21.15.90:0/763506013 shutdown_connections 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.760+0000 7f280a800640 1 -- 172.21.15.90:0/763506013 wait complete. 2024-03-31T15:56:46.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 Processor -- start 2024-03-31T15:56:46.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- start start 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28040aeb00 0x7f2804157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2804169ed0 con 0x7f280415c290 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2804169d50 con 0x7f28040aeb00 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f280416a050 con 0x7f2804072760 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41604/0 (socket says 172.21.15.90:41604) 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 -- 172.21.15.90:0/847519476 learned_addr learned my addr 172.21.15.90:0/847519476 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803400640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f27fbe00640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28040aeb00 0x7f2804157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 -- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 msgr2=0x7f2804150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 -- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28040aeb00 msgr2=0x7f2804157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28040aeb00 0x7f2804157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 -- 172.21.15.90:0/847519476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2804074d70 con 0x7f280415c290 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f2803e00640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f27fc01cd90 tx=0x7f27fc019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0b8630 con 0x7f280415c290 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc0c1070 con 0x7f280415c290 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0b89f0 con 0x7f280415c290 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28041524e0 con 0x7f280415c290 2024-03-31T15:56:46.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.764+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2804152aa0 con 0x7f280415c290 2024-03-31T15:56:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2804075880 con 0x7f280415c290 2024-03-31T15:56:46.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f27fc0b8b90 con 0x7f280415c290 2024-03-31T15:56:46.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f27fb400640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 0x7f27e006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:46.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f27fc121d50 con 0x7f280415c290 2024-03-31T15:56:46.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f2803400640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 0x7f27e006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:46.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.768+0000 7f2803400640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 0x7f27e006a870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f27f40015e0 tx=0x7f27f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:46.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.772+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f27fc0164a0 con 0x7f280415c290 2024-03-31T15:56:46.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f28040685f0 con 0x7f280415c290 2024-03-31T15:56:46.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f27fb400640 1 -- 172.21.15.90:0/847519476 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f27fc0f1aa0 con 0x7f280415c290 2024-03-31T15:56:46.969 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:46.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 msgr2=0x7f27e006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 0x7f27e006a870 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f27f40015e0 tx=0x7f27f4009040 comp rx=0 tx=0).stop 2024-03-31T15:56:46.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 msgr2=0x7f2804157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f27fc01cd90 tx=0x7f27fc019650 comp rx=0 tx=0).stop 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 shutdown_connections 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f27e00683b0 0x7f27e006a870 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f280415c290 0x7f2804157970 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28040aeb00 0x7f2804157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 --2- 172.21.15.90:0/847519476 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f2804072760 0x7f2804150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 >> 172.21.15.90:0/847519476 conn(0x7f28040b18b0 msgr2=0x7f28040b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 shutdown_connections 2024-03-31T15:56:46.972 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:46.976+0000 7f280a800640 1 -- 172.21.15.90:0/847519476 wait complete. 2024-03-31T15:56:46.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 Processor -- start 2024-03-31T15:56:47.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 -- start start 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 0x7f0db80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0db8163bd0 0x7f0db8165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0db80d0ea0 con 0x7f0db815d240 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0db80cd220 con 0x7f0db8163bd0 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbfc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0db80cf230 con 0x7f0db80ce910 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0db8163bd0 0x7f0db8165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0db8163bd0 0x7f0db8165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41608/0 (socket says 172.21.15.90:41608) 2024-03-31T15:56:47.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 -- 172.21.15.90:0/2361291756 learned_addr learned my addr 172.21.15.90:0/2361291756 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbd200640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 0x7f0db80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 -- 172.21.15.90:0/2361291756 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 msgr2=0x7f0db80cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 0x7f0db80cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.136+0000 7f0dbdc00640 1 -- 172.21.15.90:0/2361291756 >> v1:172.21.15.102:6789/0 conn(0x7f0db815d240 legacy=0x7f0db80ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0dbdc00640 1 -- 172.21.15.90:0/2361291756 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db80cd460 con 0x7f0db8163bd0 2024-03-31T15:56:47.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0dbd200640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 0x7f0db80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:47.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0dbdc00640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db8163bd0 0x7f0db8165fc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009e50 tx=0x7f0db001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535b1998b6700d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0db7400640 1 -- 172.21.15.90:0/2361291756 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00a5850 con 0x7f0db8163bd0 2024-03-31T15:56:47.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0db7400640 1 -- 172.21.15.90:0/2361291756 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db00a59f0 con 0x7f0db8163bd0 2024-03-31T15:56:47.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0db7400640 1 -- 172.21.15.90:0/2361291756 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db00a5dc0 con 0x7f0db8163bd0 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0dbfc00640 1 -- 172.21.15.90:0/2361291756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db8163bd0 msgr2=0x7f0db8165fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.140+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db8163bd0 0x7f0db8165fc0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009e50 tx=0x7f0db001c970 comp rx=0 tx=0).stop 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- 172.21.15.90:0/2361291756 shutdown_connections 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db8163bd0 0x7f0db8165fc0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/2361291756 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db80ce910 0x7f0db80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- 172.21.15.90:0/2361291756 >> 172.21.15.90:0/2361291756 conn(0x7f0db808bfd0 msgr2=0x7f0db80d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- 172.21.15.90:0/2361291756 shutdown_connections 2024-03-31T15:56:47.136 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- 172.21.15.90:0/2361291756 wait complete. 2024-03-31T15:56:47.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 Processor -- start 2024-03-31T15:56:47.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- start start 2024-03-31T15:56:47.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db815d240 0x7f0db8155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8163bd0 0x7f0db8158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0db8168d20 con 0x7f0db80ce910 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0db815d780 con 0x7f0db8163bd0 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0db8168ea0 con 0x7f0db815d240 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41618/0 (socket says 172.21.15.90:41618) 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 -- 172.21.15.90:0/3864651364 learned_addr learned my addr 172.21.15.90:0/3864651364 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 -- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db815d240 msgr2=0x7f0db8155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db815d240 0x7f0db8155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 -- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8163bd0 msgr2=0x7f0db8158ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8163bd0 0x7f0db8158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 -- 172.21.15.90:0/3864651364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db80b5bb0 con 0x7f0db80ce910 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbd200640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00c7f0 tx=0x7f0dac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac015070 con 0x7f0db80ce910 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dac007820 con 0x7f0db80ce910 2024-03-31T15:56:47.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac007c10 con 0x7f0db80ce910 2024-03-31T15:56:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.144+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db80b5090 con 0x7f0db80ce910 2024-03-31T15:56:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db80b58d0 con 0x7f0db80ce910 2024-03-31T15:56:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0db80c67b0 con 0x7f0db80ce910 2024-03-31T15:56:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0dac018070 con 0x7f0db80ce910 2024-03-31T15:56:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0db4c00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 0x7f0d9806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0dac086d40 con 0x7f0db80ce910 2024-03-31T15:56:47.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.148+0000 7f0db7e00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 0x7f0d9806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.152+0000 7f0db7e00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 0x7f0d9806a840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0da4000f70 tx=0x7f0da40079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.152+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0db80c67b0 con 0x7f0db80ce910 2024-03-31T15:56:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0db8068660 con 0x7f0db80ce910 2024-03-31T15:56:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0db4c00640 1 -- 172.21.15.90:0/3864651364 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7f0db80b5090 con 0x7f0db80ce910 2024-03-31T15:56:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:47.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 msgr2=0x7f0d9806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 0x7f0d9806a840 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0da4000f70 tx=0x7f0da40079e0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 msgr2=0x7f0db8153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00c7f0 tx=0x7f0dac00ccb0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 shutdown_connections 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d98068380 0x7f0d9806a840 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0db8163bd0 0x7f0db8158ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0db815d240 0x7f0db8155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 --2- 172.21.15.90:0/3864651364 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0db80ce910 0x7f0db8153390 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 >> 172.21.15.90:0/3864651364 conn(0x7f0db808bfd0 msgr2=0x7f0db815c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 shutdown_connections 2024-03-31T15:56:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.324+0000 7f0dbfc00640 1 -- 172.21.15.90:0/3864651364 wait complete. 2024-03-31T15:56:47.339 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:47.339 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:47.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.532+0000 7f9a6f600640 1 Processor -- start 2024-03-31T15:56:47.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.532+0000 7f9a6f600640 1 -- start start 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6f600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a6815e1b0 0x7f9a68162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6f600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6815f780 con 0x7f9a6806d5f0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6f600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f9a6806daa0 con 0x7f9a6815e1b0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6f600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9a681622b0 con 0x7f9a6806c7b0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6cc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f9a6815e1b0 0x7f9a68162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6d600640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9a6806d5f0 0x7f9a6815e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60690/0 (socket says 172.21.15.90:60690) 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6d600640 1 -- 172.21.15.90:0/2057873543 learned_addr learned my addr 172.21.15.90:0/2057873543 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a67e00640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a67400640 1 -- 172.21.15.90:0/2057873543 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241976279 0 0) 0x7f9a6815f780 con 0x7f9a6806d5f0 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6cc00640 1 -- 172.21.15.90:0/2057873543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 msgr2=0x7f9a6806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6cc00640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6cc00640 1 -- 172.21.15.90:0/2057873543 >> v1:172.21.15.90:6789/0 conn(0x7f9a6806d5f0 legacy=0x7f9a6815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a6cc00640 1 -- 172.21.15.90:0/2057873543 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6815f1c0 con 0x7f9a6815e1b0 2024-03-31T15:56:47.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.536+0000 7f9a67e00640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:47.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6cc00640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6815e1b0 0x7f9a68162740 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0097c0 tx=0x7f9a5c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ded806dcf6122cd server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a67400640 1 -- 172.21.15.90:0/2057873543 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0b1070 con 0x7f9a6815e1b0 2024-03-31T15:56:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a67400640 1 -- 172.21.15.90:0/2057873543 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5c0a8050 con 0x7f9a6815e1b0 2024-03-31T15:56:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a67400640 1 -- 172.21.15.90:0/2057873543 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0ac860 con 0x7f9a6815e1b0 2024-03-31T15:56:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 -- 172.21.15.90:0/2057873543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6815e1b0 msgr2=0x7f9a68162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6815e1b0 0x7f9a68162740 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0097c0 tx=0x7f9a5c01c3e0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 -- 172.21.15.90:0/2057873543 shutdown_connections 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2057873543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6815e1b0 0x7f9a68162740 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 -- 172.21.15.90:0/2057873543 >> 172.21.15.90:0/2057873543 conn(0x7f9a6809aa40 msgr2=0x7f9a6809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 -- 172.21.15.90:0/2057873543 shutdown_connections 2024-03-31T15:56:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.540+0000 7f9a6f600640 1 -- 172.21.15.90:0/2057873543 wait complete. 2024-03-31T15:56:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 Processor -- start 2024-03-31T15:56:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 -- start start 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6815e1b0 0x7f9a680c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9a680d2940 con 0x7f9a6806d5f0 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a680d27c0 con 0x7f9a6815e1b0 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6f600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9a680d2ac0 con 0x7f9a6806c7b0 2024-03-31T15:56:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41638/0 (socket says 172.21.15.90:41638) 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 -- 172.21.15.90:0/2016694041 learned_addr learned my addr 172.21.15.90:0/2016694041 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6cc00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a6d600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6815e1b0 0x7f9a680c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 -- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 msgr2=0x7f9a6815d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6815d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 -- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6815e1b0 msgr2=0x7f9a680c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6815e1b0 0x7f9a680c6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.544+0000 7f9a67e00640 1 -- 172.21.15.90:0/2016694041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a680c7480 con 0x7f9a6806d5f0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a67e00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e840 tx=0x7f9a5400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54017070 con 0x7f9a6806d5f0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5400c8d0 con 0x7f9a6806d5f0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a680c7760 con 0x7f9a6806d5f0 2024-03-31T15:56:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400f040 con 0x7f9a6806d5f0 2024-03-31T15:56:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a680c7d20 con 0x7f9a6806d5f0 2024-03-31T15:56:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a680686d0 con 0x7f9a6806d5f0 2024-03-31T15:56:47.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a5401a070 con 0x7f9a6806d5f0 2024-03-31T15:56:47.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 0x7f9a4806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.548+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f9a54086700 con 0x7f9a6806d5f0 2024-03-31T15:56:47.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.552+0000 7f9a6cc00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 0x7f9a4806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.552+0000 7f9a6cc00640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 0x7f9a4806a840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0ae000 tx=0x7f9a5c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.552+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f9a540564d0 con 0x7f9a6806d5f0 2024-03-31T15:56:47.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.712+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a6815e730 con 0x7f9a6806d5f0 2024-03-31T15:56:47.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.712+0000 7f9a64c00640 1 -- 172.21.15.90:0/2016694041 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f9a54084e20 con 0x7f9a6806d5f0 2024-03-31T15:56:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:47.707 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 msgr2=0x7f9a4806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 0x7f9a4806a840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0ae000 tx=0x7f9a5c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 msgr2=0x7f9a680c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e840 tx=0x7f9a5400ed00 comp rx=0 tx=0).stop 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 shutdown_connections 2024-03-31T15:56:47.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9a48068380 0x7f9a4806a840 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6815e1b0 0x7f9a680c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9a6806d5f0 0x7f9a680c66e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 --2- 172.21.15.90:0/2016694041 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f9a6806c7b0 0x7f9a6815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 >> 172.21.15.90:0/2016694041 conn(0x7f9a6809aa40 msgr2=0x7f9a6806f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 shutdown_connections 2024-03-31T15:56:47.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.716+0000 7f9a6f600640 1 -- 172.21.15.90:0/2016694041 wait complete. 2024-03-31T15:56:47.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:47.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.872+0000 7f25e0c00640 1 Processor -- start 2024-03-31T15:56:47.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.872+0000 7f25e0c00640 1 -- start start 2024-03-31T15:56:47.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25e0c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 0x7f25dc0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25e0c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25dc15e8c0 con 0x7f25dc06d460 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25e0c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25dc0d0ff0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25e0c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25dc0d12d0 con 0x7f25dc06d020 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25da800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f25dc06d460 0x7f25dc0cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60702/0 (socket says 172.21.15.90:60702) 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25da800640 1 -- 172.21.15.90:0/3918864305 learned_addr learned my addr 172.21.15.90:0/3918864305 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d3e00640 1 --2- 172.21.15.90:0/3918864305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 0x7f25dc0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399261346 0 0) 0x7f25dc15e8c0 con 0x7f25dc06d460 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c0003530 con 0x7f25dc06d460 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121101169 0 0) 0x7f25dc0d0ff0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25dc15e8c0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d3e00640 1 -- 172.21.15.90:0/3918864305 >> v1:172.21.15.102:6789/0 conn(0x7f25dc15e4e0 legacy=0x7f25dc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d3e00640 1 -- 172.21.15.90:0/3918864305 >> v1:172.21.15.90:6789/0 conn(0x7f25dc06d460 legacy=0x7f25dc0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.876+0000 7f25d3e00640 1 -- 172.21.15.90:0/3918864305 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc0cc710 con 0x7f25dc06d020 2024-03-31T15:56:47.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25d3e00640 1 --2- 172.21.15.90:0/3918864305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 0x7f25dc0ccf10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f25c40097f0 tx=0x7f25c401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0d420afea0ee992 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c40b0070 con 0x7f25dc06d020 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c40ab600 con 0x7f25dc06d020 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25d9400640 1 -- 172.21.15.90:0/3918864305 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c40ab9b0 con 0x7f25dc06d020 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- 172.21.15.90:0/3918864305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 msgr2=0x7f25dc0ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3918864305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 0x7f25dc0ccf10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f25c40097f0 tx=0x7f25c401c450 comp rx=0 tx=0).stop 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- 172.21.15.90:0/3918864305 shutdown_connections 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3918864305 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d020 0x7f25dc0ccf10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- 172.21.15.90:0/3918864305 >> 172.21.15.90:0/3918864305 conn(0x7f25dc0d1f20 msgr2=0x7f25dc0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- 172.21.15.90:0/3918864305 shutdown_connections 2024-03-31T15:56:47.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- 172.21.15.90:0/3918864305 wait complete. 2024-03-31T15:56:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 Processor -- start 2024-03-31T15:56:47.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.880+0000 7f25e0c00640 1 -- start start 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25dc06d020 0x7f25dc15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d460 0x7f25dc154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc0d0d90 con 0x7f25dc15e4e0 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc0d0c10 con 0x7f25dc06d020 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f25dc16c040 con 0x7f25dc06d460 2024-03-31T15:56:47.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41646/0 (socket says 172.21.15.90:41646) 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 -- 172.21.15.90:0/3759977795 learned_addr learned my addr 172.21.15.90:0/3759977795 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d9e00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25dc06d020 0x7f25dc15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d3e00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d460 0x7f25dc154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 -- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d460 msgr2=0x7f25dc154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d460 0x7f25dc154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 -- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25dc06d020 msgr2=0x7f25dc15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25dc06d020 0x7f25dc15be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 -- 172.21.15.90:0/3759977795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc15ae00 con 0x7f25dc15e4e0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25da800640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f25d400cb10 tx=0x7f25d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4017070 con 0x7f25dc15e4e0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d400ac70 con 0x7f25dc15e4e0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc15b0a0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4012620 con 0x7f25dc15e4e0 2024-03-31T15:56:47.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc15b660 con 0x7f25dc15e4e0 2024-03-31T15:56:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25dc0685f0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.884+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25d401a070 con 0x7f25dc15e4e0 2024-03-31T15:56:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.888+0000 7f25d2000640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 0x7f25b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.888+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f25d40867b0 con 0x7f25dc15e4e0 2024-03-31T15:56:47.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.888+0000 7f25d9e00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 0x7f25b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:47.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.888+0000 7f25d9e00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 0x7f25b806a840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0011f0 tx=0x7f25cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:47.884 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:47.892+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f25d400ae20 con 0x7f25dc15e4e0 2024-03-31T15:56:48.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.048+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f25dc0cc2e0 con 0x7f25dc15e4e0 2024-03-31T15:56:48.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.048+0000 7f25d2000640 1 -- 172.21.15.90:0/3759977795 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7f25d4056620 con 0x7f25dc15e4e0 2024-03-31T15:56:48.042 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 msgr2=0x7f25b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 0x7f25b806a840 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0011f0 tx=0x7f25cc009040 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 msgr2=0x7f25dc158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f25d400cb10 tx=0x7f25d400a060 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 shutdown_connections 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f25b8068380 0x7f25b806a840 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f25dc15e4e0 0x7f25dc158ee0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f25dc06d460 0x7f25dc154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 --2- 172.21.15.90:0/3759977795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25dc06d020 0x7f25dc15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 >> 172.21.15.90:0/3759977795 conn(0x7f25dc0d1f20 msgr2=0x7f25dc155c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 shutdown_connections 2024-03-31T15:56:48.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.052+0000 7f25e0c00640 1 -- 172.21.15.90:0/3759977795 wait complete. 2024-03-31T15:56:48.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:48.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:56:48.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:56:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 Processor -- start 2024-03-31T15:56:48.210 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 -- start start 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fb006b850 0x7f7fb0067330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb00b1540 con 0x7f7fb015c460 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb00b04a0 con 0x7f7fb006bc30 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb6000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb00b0780 con 0x7f7fb006b850 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fb006b850 0x7f7fb0067330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb4c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7fb006b850 0x7f7fb0067330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41660/0 (socket says 172.21.15.90:41660) 2024-03-31T15:56:48.211 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb4c00640 1 -- 172.21.15.90:0/302295239 learned_addr learned my addr 172.21.15.90:0/302295239 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.216+0000 7f7fb4c00640 1 -- 172.21.15.90:0/302295239 >> v1:172.21.15.102:6789/0 conn(0x7f7fb006bc30 legacy=0x7f7fb015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb4c00640 1 -- 172.21.15.90:0/302295239 >> v1:172.21.15.90:6789/0 conn(0x7f7fb015c460 legacy=0x7f7fb00ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.212 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb4c00640 1 -- 172.21.15.90:0/302295239 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb00b0a60 con 0x7f7fb006b850 2024-03-31T15:56:48.214 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/302295239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb006b850 0x7f7fb0067330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009870 tx=0x7f7fa401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef5ae7deb50b4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7faf400640 1 -- 172.21.15.90:0/302295239 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40ab640 con 0x7f7fb006b850 2024-03-31T15:56:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7faf400640 1 -- 172.21.15.90:0/302295239 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa40b0070 con 0x7f7fb006b850 2024-03-31T15:56:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7faf400640 1 -- 172.21.15.90:0/302295239 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40aba00 con 0x7f7fb006b850 2024-03-31T15:56:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 -- 172.21.15.90:0/302295239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb006b850 msgr2=0x7f7fb0067330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.215 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 --2- 172.21.15.90:0/302295239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb006b850 0x7f7fb0067330 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009870 tx=0x7f7fa401c530 comp rx=0 tx=0).stop 2024-03-31T15:56:48.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 -- 172.21.15.90:0/302295239 shutdown_connections 2024-03-31T15:56:48.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 --2- 172.21.15.90:0/302295239 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb006b850 0x7f7fb0067330 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 -- 172.21.15.90:0/302295239 >> 172.21.15.90:0/302295239 conn(0x7f7fb00b1fd0 msgr2=0x7f7fb00b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 -- 172.21.15.90:0/302295239 shutdown_connections 2024-03-31T15:56:48.216 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.220+0000 7f7fb6000640 1 -- 172.21.15.90:0/302295239 wait complete. 2024-03-31T15:56:48.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 Processor -- start 2024-03-31T15:56:48.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- start start 2024-03-31T15:56:48.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 0x7f7fb0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 0x7f7fb015e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb0073b50 con 0x7f7fb015c460 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb00739d0 con 0x7f7fb006bc30 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb0067870 con 0x7f7fb006b850 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41674/0 (socket says 172.21.15.90:41674) 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 -- 172.21.15.90:0/1470152198 learned_addr learned my addr 172.21.15.90:0/1470152198 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 0x7f7fb0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fafe00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 0x7f7fb015e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 -- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 msgr2=0x7f7fb0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 0x7f7fb0156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 -- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 msgr2=0x7f7fb015e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.220 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 0x7f7fb015e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 -- 172.21.15.90:0/1470152198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb015e8d0 con 0x7f7fb015c460 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 0x7f7fb0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fafe00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 0x7f7fb015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb5600640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c7a0 tx=0x7f7fa800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8003b80 con 0x7f7fb015c460 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa8003d20 con 0x7f7fb015c460 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb015ebb0 con 0x7f7fb015c460 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa800aa30 con 0x7f7fb015c460 2024-03-31T15:56:48.221 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.224+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb015f170 con 0x7f7fb015c460 2024-03-31T15:56:48.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb0068d00 con 0x7f7fb015c460 2024-03-31T15:56:48.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7fa800abd0 con 0x7f7fb015c460 2024-03-31T15:56:48.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7facc00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 0x7f7f9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f7fa8086a70 con 0x7f7fb015c460 2024-03-31T15:56:48.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 0x7f7f9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.228+0000 7f7fb4c00640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 0x7f7f9006a840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40af000 tx=0x7f7fa4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.232+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7fa8056e90 con 0x7f7fb015c460 2024-03-31T15:56:48.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.444+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7fb00b1980 con 0x7f7fb015c460 2024-03-31T15:56:48.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.444+0000 7f7facc00640 1 -- 172.21.15.90:0/1470152198 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7f7fa8056790 con 0x7f7fb015c460 2024-03-31T15:56:48.439 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 msgr2=0x7f7f9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 0x7f7f9006a840 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40af000 tx=0x7f7fa4008040 comp rx=0 tx=0).stop 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 msgr2=0x7f7fb00cf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7fa800c7a0 tx=0x7f7fa800cc60 comp rx=0 tx=0).stop 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 shutdown_connections 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7f90068380 0x7f7f9006a840 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7fb015c460 0x7f7fb00cf8c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fb006bc30 0x7f7fb015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 --2- 172.21.15.90:0/1470152198 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7fb006b850 0x7f7fb0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 >> 172.21.15.90:0/1470152198 conn(0x7f7fb00b1fd0 msgr2=0x7f7fb006ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 shutdown_connections 2024-03-31T15:56:48.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.448+0000 7f7fb6000640 1 -- 172.21.15.90:0/1470152198 wait complete. 2024-03-31T15:56:48.460 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:48.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 Processor -- start 2024-03-31T15:56:48.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 -- start start 2024-03-31T15:56:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f16ac15e4e0 0x7f16ac06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f16ac06ce50 0x7f16ac06d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16ac15e8c0 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f16ac0d0ff0 con 0x7f16ac06ce50 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.652+0000 7f16b1a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f16ac0d12d0 con 0x7f16ac15e4e0 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f16ac06ce50 0x7f16ac06d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f16ac06ce50 0x7f16ac06d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41680/0 (socket says 172.21.15.90:41680) 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 -- 172.21.15.90:0/1175274841 learned_addr learned my addr 172.21.15.90:0/1175274841 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:48.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16aaa00640 1 --2- 172.21.15.90:0/1175274841 >> v2:172.21.15.102:3300/0 conn(0x7f16ac15e4e0 0x7f16ac06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 -- 172.21.15.90:0/1175274841 >> v2:172.21.15.102:3300/0 conn(0x7f16ac15e4e0 msgr2=0x7f16ac06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 --2- 172.21.15.90:0/1175274841 >> v2:172.21.15.102:3300/0 conn(0x7f16ac15e4e0 0x7f16ac06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 -- 172.21.15.90:0/1175274841 >> v1:172.21.15.102:6789/0 conn(0x7f16ac0ce7a0 legacy=0x7f16ac0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 -- 172.21.15.90:0/1175274841 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac0cc710 con 0x7f16ac06ce50 2024-03-31T15:56:48.653 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16a3e00640 1 --2- 172.21.15.90:0/1175274841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac06ce50 0x7f16ac06d2c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1694009870 tx=0x7f169401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55263ff72fcb81fc server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16aa000640 1 -- 172.21.15.90:0/1175274841 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940b1070 con 0x7f16ac06ce50 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16aa000640 1 -- 172.21.15.90:0/1175274841 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16940ac600 con 0x7f16ac06ce50 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.656+0000 7f16aa000640 1 -- 172.21.15.90:0/1175274841 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940ac9d0 con 0x7f16ac06ce50 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- 172.21.15.90:0/1175274841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac06ce50 msgr2=0x7f16ac06d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- 172.21.15.90:0/1175274841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac06ce50 0x7f16ac06d2c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1694009870 tx=0x7f169401c490 comp rx=0 tx=0).stop 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- 172.21.15.90:0/1175274841 shutdown_connections 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- 172.21.15.90:0/1175274841 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac06ce50 0x7f16ac06d2c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- 172.21.15.90:0/1175274841 >> v2:172.21.15.102:3300/0 conn(0x7f16ac15e4e0 0x7f16ac06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- 172.21.15.90:0/1175274841 >> 172.21.15.90:0/1175274841 conn(0x7f16ac0d1f20 msgr2=0x7f16ac0d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- 172.21.15.90:0/1175274841 shutdown_connections 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- 172.21.15.90:0/1175274841 wait complete. 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 Processor -- start 2024-03-31T15:56:48.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- start start 2024-03-31T15:56:48.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac06ce50 0x7f16ac155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f16ac15e4e0 0x7f16ac06f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac0d0d90 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac0d0c10 con 0x7f16ac06ce50 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16b1a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f16ac16b840 con 0x7f16ac15e4e0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41686/0 (socket says 172.21.15.90:41686) 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 -- 172.21.15.90:0/2316388089 learned_addr learned my addr 172.21.15.90:0/2316388089 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16aaa00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac06ce50 0x7f16ac155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16ab400640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f16ac15e4e0 0x7f16ac06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 -- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f16ac15e4e0 msgr2=0x7f16ac06f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f16ac15e4e0 0x7f16ac06f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 -- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac06ce50 msgr2=0x7f16ac155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac06ce50 0x7f16ac155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.660+0000 7f16a3e00640 1 -- 172.21.15.90:0/2316388089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac156290 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a3e00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f169401a0c0 tx=0x7f169401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940bc070 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16940b7430 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940b1040 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ac09f250 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ac09f810 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ac0685f0 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f16940b63d0 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 0x7f168806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f16941200a0 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.664+0000 7f16aaa00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 0x7f168806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.668+0000 7f16aaa00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 0x7f168806a840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16ac06dd90 tx=0x7f169c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.668+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1694016110 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16ac070830 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16a2a00640 1 -- 172.21.15.90:0/2316388089 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f16940effa0 con 0x7f16ac0ce7a0 2024-03-31T15:56:48.821 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 msgr2=0x7f168806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 0x7f168806a840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16ac06dd90 tx=0x7f169c009190 comp rx=0 tx=0).stop 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 msgr2=0x7f16ac09e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f169401a0c0 tx=0x7f169401acd0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 shutdown_connections 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1688068380 0x7f168806a840 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f16ac15e4e0 0x7f16ac06f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f16ac0ce7a0 0x7f16ac09e7e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 --2- 172.21.15.90:0/2316388089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16ac06ce50 0x7f16ac155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 >> 172.21.15.90:0/2316388089 conn(0x7f16ac0d1f20 msgr2=0x7f16ac0c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 shutdown_connections 2024-03-31T15:56:48.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.828+0000 7f16b1a00640 1 -- 172.21.15.90:0/2316388089 wait complete. 2024-03-31T15:56:48.842 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:48.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 Processor -- start 2024-03-31T15:56:48.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 -- start start 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c20164be0 0x7f7c20166fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c200d0ea0 con 0x7f7c2015d240 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c200cd460 con 0x7f7c200ce910 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c28000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7c200cd220 con 0x7f7c20164be0 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c24c00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7c2015d240 0x7f7c200ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60716/0 (socket says 172.21.15.90:60716) 2024-03-31T15:56:48.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c24c00640 1 -- 172.21.15.90:0/3415320926 learned_addr learned my addr 172.21.15.90:0/3415320926 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> v2:172.21.15.102:3300/0 conn(0x7f7c20164be0 0x7f7c20166fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003331807 0 0) 0x7f7c200d0ea0 con 0x7f7c2015d240 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.984+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c08003530 con 0x7f7c2015d240 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582098617 0 0) 0x7f7c08003530 con 0x7f7c2015d240 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 >> v2:172.21.15.102:3300/0 conn(0x7f7c20164be0 msgr2=0x7f7c20166fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:48.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 --2- 172.21.15.90:0/3415320926 >> v2:172.21.15.102:3300/0 conn(0x7f7c20164be0 0x7f7c20166fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 >> v1:172.21.15.102:6789/0 conn(0x7f7c200ce910 legacy=0x7f7c200cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c200cd740 con 0x7f7c2015d240 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f7c0c001680 con 0x7f7c2015d240 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 >> v1:172.21.15.90:6789/0 conn(0x7f7c2015d240 legacy=0x7f7c200ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c080039c0 0x7f7c08003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c080043c0 0x7f7c08004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 0x7f7c0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c200cd220 con 0x7f7c08005130 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0800c070 con 0x7f7c080039c0 2024-03-31T15:56:48.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0800c270 con 0x7f7c080043c0 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> v2:172.21.15.102:3300/0 conn(0x7f7c20164be0 0x7f7c20166fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 0x7f7c0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c24c00640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c080043c0 0x7f7c08004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c25600640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c080039c0 0x7f7c08003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 -- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c080043c0 msgr2=0x7f7c08004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c080043c0 0x7f7c08004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 -- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c080039c0 msgr2=0x7f7c08003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c080039c0 0x7f7c08003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 -- 172.21.15.90:0/3415320926 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c08003530 con 0x7f7c08005130 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.988+0000 7f7c26000640 1 -- 172.21.15.90:0/3415320926 reap_dead start 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c26000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 0x7f7c0800b9d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800e320 tx=0x7f7c1801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1d6e658e007d0e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180b2070 con 0x7f7c08005130 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c180a5e50 con 0x7f7c08005130 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c1fe00640 1 -- 172.21.15.90:0/3415320926 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18009de0 con 0x7f7c08005130 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 msgr2=0x7f7c0800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 0x7f7c0800b9d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800e320 tx=0x7f7c1801ce40 comp rx=0 tx=0).stop 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- 172.21.15.90:0/3415320926 shutdown_connections 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 --2- 172.21.15.90:0/3415320926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c08005130 0x7f7c0800b9d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- 172.21.15.90:0/3415320926 >> 172.21.15.90:0/3415320926 conn(0x7f7c2008bfd0 msgr2=0x7f7c200d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- 172.21.15.90:0/3415320926 shutdown_connections 2024-03-31T15:56:48.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- 172.21.15.90:0/3415320926 wait complete. 2024-03-31T15:56:48.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 Processor -- start 2024-03-31T15:56:48.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- start start 2024-03-31T15:56:48.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 0x7f7c20155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c201565c0 0x7f7c20163a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2015d780 con 0x7f7c200ca630 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7c200cdd00 con 0x7f7c08005130 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c28000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7c200cecf0 con 0x7f7c201565c0 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c26000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c25600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 0x7f7c20155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c26000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41708/0 (socket says 172.21.15.90:41708) 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c26000640 1 -- 172.21.15.90:0/763622434 learned_addr learned my addr 172.21.15.90:0/763622434 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.992+0000 7f7c24c00640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c201565c0 0x7f7c20163a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 -- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c201565c0 msgr2=0x7f7c20163a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c201565c0 0x7f7c20163a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 -- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 msgr2=0x7f7c20155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 0x7f7c20155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 -- 172.21.15.90:0/763622434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c201627c0 con 0x7f7c200ca630 2024-03-31T15:56:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c25600640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 0x7f7c20155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c26000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7c1801cc70 tx=0x7f7c18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180b2040 con 0x7f7c200ca630 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c180a5c50 con 0x7f7c200ca630 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18009de0 con 0x7f7c200ca630 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c20162aa0 con 0x7f7c200ca630 2024-03-31T15:56:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c20163060 con 0x7f7c200ca630 2024-03-31T15:56:48.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c20068660 con 0x7f7c200ca630 2024-03-31T15:56:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7c180bd030 con 0x7f7c200ca630 2024-03-31T15:56:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 0x7f7c0006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:48.996+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f7c18120fe0 con 0x7f7c200ca630 2024-03-31T15:56:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.000+0000 7f7c25600640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 0x7f7c0006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:48.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.000+0000 7f7c25600640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 0x7f7c0006a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c200ca490 tx=0x7f7c1400acf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:49.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.008+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7c18016b50 con 0x7f7c200ca630 2024-03-31T15:56:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.176+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7c200ce6f0 con 0x7f7c200ca630 2024-03-31T15:56:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.176+0000 7f7c1d600640 1 -- 172.21.15.90:0/763622434 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7f7c180f0e10 con 0x7f7c200ca630 2024-03-31T15:56:49.171 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 msgr2=0x7f7c0006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 0x7f7c0006a870 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7c200ca490 tx=0x7f7c1400acf0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 msgr2=0x7f7c200caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7c1801cc70 tx=0x7f7c18009750 comp rx=0 tx=0).stop 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 shutdown_connections 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7c000683b0 0x7f7c0006a870 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7c200ca630 0x7f7c200caac0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7c201565c0 0x7f7c20163a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 --2- 172.21.15.90:0/763622434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c08005130 0x7f7c20155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 >> 172.21.15.90:0/763622434 conn(0x7f7c2008bfd0 msgr2=0x7f7c200b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 shutdown_connections 2024-03-31T15:56:49.174 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.180+0000 7f7c28000640 1 -- 172.21.15.90:0/763622434 wait complete. 2024-03-31T15:56:49.192 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:56:49.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:49.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42032]: fuse finished with error 0 and tester_r 0 2024-03-31T15:56:49.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:56:49.249 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:49.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.249 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:56:49.304 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-31T15:56:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.488+0000 7f0d1ca00640 1 Processor -- start 2024-03-31T15:56:49.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.488+0000 7f0d1ca00640 1 -- start start 2024-03-31T15:56:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.488+0000 7f0d1ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1815beb0 0x7f0d1815f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.488+0000 7f0d1ca00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d1806edb0 0x7f0d1806b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d1ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d18162170 con 0x7f0d1806d4d0 2024-03-31T15:56:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d1ca00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f0d18162450 con 0x7f0d1806edb0 2024-03-31T15:56:49.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d1ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d18162730 con 0x7f0d1815beb0 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d1806edb0 0x7f0d1806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f0d1806edb0 0x7f0d1806b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41722/0 (socket says 172.21.15.90:41722) 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 -- 172.21.15.90:0/3603040074 learned_addr learned my addr 172.21.15.90:0/3603040074 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d15e00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1815beb0 0x7f0d1815f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 -- 172.21.15.90:0/3603040074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1815beb0 msgr2=0x7f0d1815f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1815beb0 0x7f0d1815f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 -- 172.21.15.90:0/3603040074 >> v1:172.21.15.102:6789/0 conn(0x7f0d1806d4d0 legacy=0x7f0d180ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.485 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 -- 172.21.15.90:0/3603040074 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d18162a10 con 0x7f0d1806edb0 2024-03-31T15:56:49.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d0fe00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1806edb0 0x7f0d1806b820 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009870 tx=0x7f0d0001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5ab7d5c84ec758 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:49.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d15400640 1 -- 172.21.15.90:0/3603040074 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d000b0070 con 0x7f0d1806edb0 2024-03-31T15:56:49.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d15400640 1 -- 172.21.15.90:0/3603040074 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d000ab600 con 0x7f0d1806edb0 2024-03-31T15:56:49.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.492+0000 7f0d15400640 1 -- 172.21.15.90:0/3603040074 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d000ab9d0 con 0x7f0d1806edb0 2024-03-31T15:56:49.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- 172.21.15.90:0/3603040074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1806edb0 msgr2=0x7f0d1806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1806edb0 0x7f0d1806b820 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009870 tx=0x7f0d0001c440 comp rx=0 tx=0).stop 2024-03-31T15:56:49.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- 172.21.15.90:0/3603040074 shutdown_connections 2024-03-31T15:56:49.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1806edb0 0x7f0d1806b820 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.488 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/3603040074 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1815beb0 0x7f0d1815f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- 172.21.15.90:0/3603040074 >> 172.21.15.90:0/3603040074 conn(0x7f0d180b1c80 msgr2=0x7f0d180b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:49.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- 172.21.15.90:0/3603040074 shutdown_connections 2024-03-31T15:56:49.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- 172.21.15.90:0/3603040074 wait complete. 2024-03-31T15:56:49.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 Processor -- start 2024-03-31T15:56:49.490 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- start start 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1806d4d0 0x7f0d181551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1806edb0 0x7f0d1815f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1816b1a0 con 0x7f0d1815beb0 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1806c170 con 0x7f0d1806edb0 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d1ca00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1816b320 con 0x7f0d1806d4d0 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41734/0 (socket says 172.21.15.90:41734) 2024-03-31T15:56:49.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 -- 172.21.15.90:0/1947126019 learned_addr learned my addr 172.21.15.90:0/1947126019 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d0fe00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1806edb0 0x7f0d1815f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d15e00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1806d4d0 0x7f0d181551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 -- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1806d4d0 msgr2=0x7f0d181551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1806d4d0 0x7f0d181551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 -- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1806edb0 msgr2=0x7f0d1815f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1806edb0 0x7f0d1815f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.496+0000 7f0d16800640 1 -- 172.21.15.90:0/1947126019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1815f6c0 con 0x7f0d1815beb0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d16800640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c730 tx=0x7f0d1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10003b80 con 0x7f0d1815beb0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d10003d20 con 0x7f0d1815beb0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1815f9c0 con 0x7f0d1815beb0 2024-03-31T15:56:49.493 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1000aa30 con 0x7f0d1815beb0 2024-03-31T15:56:49.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d18160200 con 0x7f0d1815beb0 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d180686d0 con 0x7f0d1815beb0 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d1000abd0 con 0x7f0d1815beb0 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 0x7f0cf406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0d100868d0 con 0x7f0d1815beb0 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.500+0000 7f0d15e00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 0x7f0cf406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:49.495 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.504+0000 7f0d15e00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 0x7f0cf406a840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009ac0 tx=0x7f0d08009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:49.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.504+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d1815f9c0 con 0x7f0d1815beb0 2024-03-31T15:56:49.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0d18074390 con 0x7f0d1815beb0 2024-03-31T15:56:49.693 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d0e000640 1 -- 172.21.15.90:0/1947126019 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+73 (secure 0 0 0) 0x7f0d100566a0 con 0x7f0d1815beb0 2024-03-31T15:56:49.693 INFO:teuthology.orchestra.run.smithi090.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-31T15:56:49.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 msgr2=0x7f0cf406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 0x7f0cf406a840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009ac0 tx=0x7f0d08009190 comp rx=0 tx=0).stop 2024-03-31T15:56:49.695 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 msgr2=0x7f0d180b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000c730 tx=0x7f0d1000cbf0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 shutdown_connections 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0cf4068380 0x7f0cf406a840 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d1815beb0 0x7f0d180b0e90 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d1806edb0 0x7f0d1815f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 --2- 172.21.15.90:0/1947126019 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d1806d4d0 0x7f0d181551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 >> 172.21.15.90:0/1947126019 conn(0x7f0d180b1c80 msgr2=0x7f0d180ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 shutdown_connections 2024-03-31T15:56:49.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49.700+0000 7f0d1ca00640 1 -- 172.21.15.90:0/1947126019 wait complete. 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-31T15:56:49.714 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:49.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.714 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42314sec preferred_lft 42314sec 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 02:42:88:55:26:31 brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.725 INFO:teuthology.orchestra.run.smithi090.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-31T15:56:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-03-31T15:56:49.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.727 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-31T15:56:49.784 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-31T15:56:49.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.787 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-31T15:56:49.839 INFO:teuthology.orchestra.run.smithi090.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:49.840 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:49.840 INFO:teuthology.orchestra.run.smithi090.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-03-31T15:56:49.840 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-31T15:56:49.840 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-03-31T15:56:49.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-31T15:56:49.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:49.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.973 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:56:49.981 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:56:49.981 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:56:49.982 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:49.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:49.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-03-31T15:56:50.045 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-31T15:56:50.045 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:50.046 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:50.099 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-31T15:56:50.099 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-31T15:56:50.100 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:50.155 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:50.226 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:50.276 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:50.277 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:50.278 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:50.279 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:50.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:50.280 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:50.331 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-31T15:56:50.331 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-31T15:56:50.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-03-31T15:56:50.407 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-03-31T15:56:50.467 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/run/lock 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/fs/nfsd 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-03-31T15:56:50.468 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysusers.service 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/rpc_pipefs 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-0 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-1 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-2 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph/osd/ceph-3 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:50.469 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:56:50.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:50.470 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-03-31T15:56:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-03-31T15:56:50.479+0000 7efcd5803580 -1 init, newargv = 0x55dae02591f0 newargc=15 2024-03-31T15:56:50.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42432]: starting ceph client 2024-03-31T15:56:50.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42432]: starting fuse 2024-03-31T15:56:50.483 INFO:teuthology.orchestra.run.smithi090.stdout:43 2024-03-31T15:56:50.483 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-31T15:56:50.483 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-03-31T15:56:50.483 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-03-31T15:56:50.483 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:50.484 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-03-31T15:56:50.485 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:56:50.607 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.42432.asok 2024-03-31T15:56:50.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-31T15:56:50.618 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42432.asok 2024-03-31T15:56:50.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:50.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42432.asok status 2024-03-31T15:56:50.786 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:56:50.786 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "d2271579d5e735cfd6d436b8349c004f06527cb0", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.0.0-2574-gd2271579 (d2271579d5e735cfd6d436b8349c004f06527cb0) squid (dev)", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "42432", 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 6558, 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-03-31T15:56:50.787 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 6558 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1217475348 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 1217475348 2024-03-31T15:56:50.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.6558 192.168.0.1:0/1217475348", 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/1217475348", 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 27, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 55, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "cephfs" 2024-03-31T15:56:50.789 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:56:50.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:50.801 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-03-31T15:56:50.801 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:50.810 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-03-31T15:56:50.811 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:50.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:50.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> stdin-killer -- python3 -c ' 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> import errno 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-31T15:56:50.872 DEBUG:teuthology.orchestra.run.smithi090:> try: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> if True: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> ret = os.read(fd, 1024) 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> else: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> os.write(fd, b'"'"'content'"'"') 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> except OSError as e: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> if e.errno != errno.EPERM: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> raise 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> else: 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("client does not check permission of data pool") 2024-03-31T15:56:50.873 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-03-31T15:56:50.981 INFO:teuthology.orchestra.run.smithi090.stderr:Traceback (most recent call last): 2024-03-31T15:56:50.981 INFO:teuthology.orchestra.run.smithi090.stderr: File "", line 15, in 2024-03-31T15:56:50.981 INFO:teuthology.orchestra.run.smithi090.stderr:RuntimeError: client does not check permission of data pool 2024-03-31T15:56:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:50 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-31T15:56:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:50.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-31T15:56:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 Processor -- start 2024-03-31T15:56:51.181 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 -- start start 2024-03-31T15:56:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f776015c320 0x7f77600cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 0x7f77600af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77600cefc0 con 0x7f77600d2b20 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f77600aed70 con 0x7f776015c320 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7765200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f77600cf2a0 con 0x7f77600cd640 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775de00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f776015c320 0x7f77600cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7757e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f77600d2b20 0x7f77600ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60722/0 (socket says 172.21.15.90:60722) 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f7757e00640 1 -- 172.21.15.90:0/3529942612 learned_addr learned my addr 172.21.15.90:0/3529942612 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775e800640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 0x7f77600af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775de00640 1 -- 172.21.15.90:0/3529942612 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 msgr2=0x7f77600af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775de00640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 0x7f77600af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775de00640 1 -- 172.21.15.90:0/3529942612 >> v1:172.21.15.90:6789/0 conn(0x7f77600d2b20 legacy=0x7f77600ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775de00640 1 -- 172.21.15.90:0/3529942612 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77600cf580 con 0x7f776015c320 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775d400640 1 -- 172.21.15.90:0/3529942612 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584377480 0 0) 0x7f77600cefc0 con 0x7f77600d2b20 2024-03-31T15:56:51.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.187+0000 7f775e800640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 0x7f77600af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f775de00640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f776015c320 0x7f77600cc9f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f77500097c0 tx=0x7f775001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0594f966d8e6b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f775d400640 1 -- 172.21.15.90:0/3529942612 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500b0070 con 0x7f776015c320 2024-03-31T15:56:51.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f775d400640 1 -- 172.21.15.90:0/3529942612 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77600d0f20 con 0x7f776015c320 2024-03-31T15:56:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f775d400640 1 -- 172.21.15.90:0/3529942612 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500ab820 con 0x7f776015c320 2024-03-31T15:56:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 -- 172.21.15.90:0/3529942612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f776015c320 msgr2=0x7f77600cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f776015c320 0x7f77600cc9f0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f77500097c0 tx=0x7f775001c410 comp rx=0 tx=0).stop 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 -- 172.21.15.90:0/3529942612 shutdown_connections 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f77600cd640 0x7f77600af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 --2- 172.21.15.90:0/3529942612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f776015c320 0x7f77600cc9f0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 -- 172.21.15.90:0/3529942612 >> 172.21.15.90:0/3529942612 conn(0x7f77600d1f20 msgr2=0x7f77600d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 -- 172.21.15.90:0/3529942612 shutdown_connections 2024-03-31T15:56:51.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.191+0000 7f7765200640 1 -- 172.21.15.90:0/3529942612 wait complete. 2024-03-31T15:56:51.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 Processor -- start 2024-03-31T15:56:51.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 -- start start 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 0x7f77601554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 0x7f7760164f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f77600b0320 con 0x7f77600d2b20 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f77600b01a0 con 0x7f77600cd640 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7765200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f77600b04a0 con 0x7f776015c320 2024-03-31T15:56:51.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7757e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f775de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 0x7f77601554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7757e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41750/0 (socket says 172.21.15.90:41750) 2024-03-31T15:56:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f7757e00640 1 -- 172.21.15.90:0/1391174011 learned_addr learned my addr 172.21.15.90:0/1391174011 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:51.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.195+0000 7f775e800640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 0x7f7760164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 -- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 msgr2=0x7f77601554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.193 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 0x7f77601554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 -- 172.21.15.90:0/1391174011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 msgr2=0x7f77601649f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 -- 172.21.15.90:0/1391174011 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776006d780 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7757e00640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775de00640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 0x7f77601554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775e800640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 0x7f7760164f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7758007d80 tx=0x7f775800d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775801a070 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7758015400 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7760165560 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77580157d0 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7760165b20 con 0x7f776015c320 2024-03-31T15:56:51.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77600685f0 con 0x7f776015c320 2024-03-31T15:56:51.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f775801d070 con 0x7f776015c320 2024-03-31T15:56:51.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 0x7f773c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f7758086c40 con 0x7f776015c320 2024-03-31T15:56:51.195 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.199+0000 7f775de00640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 0x7f773c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.196 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.203+0000 7f775de00640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 0x7f773c06a840 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f77500ad000 tx=0x7f77500a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.199 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.207+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7758010070 con 0x7f776015c320 2024-03-31T15:56:51.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.419+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f776015cfa0 con 0x7f776015c320 2024-03-31T15:56:51.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.423+0000 7f7756000640 1 -- 172.21.15.90:0/1391174011 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v20) ==== 184+0+141 (secure 0 0 0) 0x7f7758056a10 con 0x7f776015c320 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDShglmNpj1DBAAFqyu96HBWKVqXF4ltG833A== 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stdout: caps mds = "allow" 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stdout: caps mon = "allow r" 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-31T15:56:51.419 INFO:teuthology.orchestra.run.smithi090.stderr:updated caps for client.0 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 msgr2=0x7f773c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 0x7f773c06a840 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f77500ad000 tx=0x7f77500a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 msgr2=0x7f7760164f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 0x7f7760164f30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7758007d80 tx=0x7f775800d4a0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 shutdown_connections 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f773c068380 0x7f773c06a840 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f776015c320 0x7f7760164f30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f77600d2b20 0x7f77601649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 --2- 172.21.15.90:0/1391174011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77600cd640 0x7f77601554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 >> 172.21.15.90:0/1391174011 conn(0x7f77600d1f20 msgr2=0x7f77600b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 shutdown_connections 2024-03-31T15:56:51.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.427+0000 7f7765200640 1 -- 172.21.15.90:0/1391174011 wait complete. 2024-03-31T15:56:51.440 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-31T15:56:51.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:51.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-31T15:56:51.591 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-31T15:56:51.591 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:51.591 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:51.592 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:51.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:51.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.783+0000 7f6ea9c00640 1 Processor -- start 2024-03-31T15:56:51.778 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.783+0000 7f6ea9c00640 1 -- start start 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 0x7f6ea4165c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea415f1d0 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea415f4b0 con 0x7f6ea415bfb0 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea40cb1a0 con 0x7f6ea4072840 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6ea40ccaf0 0x7f6ea415e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60738/0 (socket says 172.21.15.90:60738) 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3e00640 1 -- 172.21.15.90:0/3545832039 learned_addr learned my addr 172.21.15.90:0/3545832039 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:51.781 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9200640 1 --2- 172.21.15.90:0/3545832039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 0x7f6ea4165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3924460967 0 0) 0x7f6ea415f1d0 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e88003530 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110091832 0 0) 0x7f6ea415f4b0 con 0x7f6ea415bfb0 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea415f1d0 con 0x7f6ea415bfb0 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9200640 1 -- 172.21.15.90:0/3545832039 >> v1:172.21.15.102:6789/0 conn(0x7f6ea415bfb0 legacy=0x7f6ea4072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9200640 1 -- 172.21.15.90:0/3545832039 >> v1:172.21.15.90:6789/0 conn(0x7f6ea40ccaf0 legacy=0x7f6ea415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.787+0000 7f6ea9200640 1 -- 172.21.15.90:0/3545832039 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea415f790 con 0x7f6ea4072840 2024-03-31T15:56:51.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9200640 1 --2- 172.21.15.90:0/3545832039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 0x7f6ea4165c60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009dc0 tx=0x7f6e9c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5903843187aaecb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0b2070 con 0x7f6ea4072840 2024-03-31T15:56:51.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9c0a5940 con 0x7f6ea4072840 2024-03-31T15:56:51.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea3400640 1 -- 172.21.15.90:0/3545832039 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0a5d10 con 0x7f6ea4072840 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 -- 172.21.15.90:0/3545832039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 msgr2=0x7f6ea4165c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/3545832039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 0x7f6ea4165c60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009dc0 tx=0x7f6e9c01c990 comp rx=0 tx=0).stop 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 -- 172.21.15.90:0/3545832039 shutdown_connections 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/3545832039 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea4072840 0x7f6ea4165c60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 -- 172.21.15.90:0/3545832039 >> 172.21.15.90:0/3545832039 conn(0x7f6ea40cd660 msgr2=0x7f6ea40cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 -- 172.21.15.90:0/3545832039 shutdown_connections 2024-03-31T15:56:51.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.791+0000 7f6ea9c00640 1 -- 172.21.15.90:0/3545832039 wait complete. 2024-03-31T15:56:51.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 Processor -- start 2024-03-31T15:56:51.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 -- start start 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea4072840 0x7f6ea414bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea415bfb0 0x7f6ea40ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4070be0 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4070a60 con 0x7f6ea4072840 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea41689c0 con 0x7f6ea415bfb0 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41760/0 (socket says 172.21.15.90:41760) 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 -- 172.21.15.90:0/452695475 learned_addr learned my addr 172.21.15.90:0/452695475 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea9200640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea415bfb0 0x7f6ea40ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea3e00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea4072840 0x7f6ea414bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 -- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea415bfb0 msgr2=0x7f6ea40ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea415bfb0 0x7f6ea40ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 -- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea4072840 msgr2=0x7f6ea414bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea4072840 0x7f6ea414bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 -- 172.21.15.90:0/452695475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea414d680 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6e9be00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c490 tx=0x7f6e8c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c0052d0 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.795+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8c005470 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea414d960 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c012870 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea414df20 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea4068d00 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6e8c012a10 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea0c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:51.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f6e8c086720 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.799+0000 7f6ea3e00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:51.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.803+0000 7f6ea3e00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6e94007fd0 tx=0x7f6e94000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:51.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.803+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6e8c056460 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ea4073490 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea0c00640 1 -- 172.21.15.90:0/452695475 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f6e8c05a760 con 0x7f6ea40ccaf0 2024-03-31T15:56:51.957 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:51.957 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:51.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 msgr2=0x7f6e8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6e94007fd0 tx=0x7f6e94000bd0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 msgr2=0x7f6ea415d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:51.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c490 tx=0x7f6e8c00c950 comp rx=0 tx=0).stop 2024-03-31T15:56:51.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 shutdown_connections 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6e80068380 0x7f6e8006a840 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6ea415bfb0 0x7f6ea40ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6ea40ccaf0 0x7f6ea415d2f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 --2- 172.21.15.90:0/452695475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea4072840 0x7f6ea414bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 >> 172.21.15.90:0/452695475 conn(0x7f6ea40cd660 msgr2=0x7f6ea4072130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 shutdown_connections 2024-03-31T15:56:51.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:51.963+0000 7f6ea9c00640 1 -- 172.21.15.90:0/452695475 wait complete. 2024-03-31T15:56:51.980 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:51.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:52.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 Processor -- start 2024-03-31T15:56:52.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 -- start start 2024-03-31T15:56:52.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2540b15f0 0x7ff25406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2540d2a20 0x7ff25409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff25409b390 con 0x7ff25409af80 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff2540d0e40 con 0x7ff2540b15f0 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff25bc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff2540cff90 con 0x7ff2540d2a20 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2540b15f0 0x7ff25406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff2540b15f0 0x7ff25406d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41762/0 (socket says 172.21.15.90:41762) 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 -- 172.21.15.90:0/3145754949 learned_addr learned my addr 172.21.15.90:0/3145754949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff24be00640 1 --2- 172.21.15.90:0/3145754949 >> v2:172.21.15.102:3300/0 conn(0x7ff2540d2a20 0x7ff25409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 -- 172.21.15.90:0/3145754949 >> v2:172.21.15.102:3300/0 conn(0x7ff2540d2a20 msgr2=0x7ff25409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 --2- 172.21.15.90:0/3145754949 >> v2:172.21.15.102:3300/0 conn(0x7ff2540d2a20 0x7ff25409aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 -- 172.21.15.90:0/3145754949 >> v1:172.21.15.90:6789/0 conn(0x7ff25409af80 legacy=0x7ff254162080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.127+0000 7ff259200640 1 -- 172.21.15.90:0/3145754949 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540d0230 con 0x7ff2540b15f0 2024-03-31T15:56:52.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff259200640 1 --2- 172.21.15.90:0/3145754949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff25406d5c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff2540ce8a0 tx=0x7ff24401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0883dd89f0b6288 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff253e00640 1 -- 172.21.15.90:0/3145754949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440b0070 con 0x7ff2540b15f0 2024-03-31T15:56:52.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff253e00640 1 -- 172.21.15.90:0/3145754949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2440ab600 con 0x7ff2540b15f0 2024-03-31T15:56:52.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff253e00640 1 -- 172.21.15.90:0/3145754949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440ab9d0 con 0x7ff2540b15f0 2024-03-31T15:56:52.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 -- 172.21.15.90:0/3145754949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 msgr2=0x7ff25406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 --2- 172.21.15.90:0/3145754949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff25406d5c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff2540ce8a0 tx=0x7ff24401c490 comp rx=0 tx=0).stop 2024-03-31T15:56:52.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 -- 172.21.15.90:0/3145754949 shutdown_connections 2024-03-31T15:56:52.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 --2- 172.21.15.90:0/3145754949 >> v2:172.21.15.102:3300/0 conn(0x7ff2540d2a20 0x7ff25409aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 --2- 172.21.15.90:0/3145754949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff25406d5c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 -- 172.21.15.90:0/3145754949 >> 172.21.15.90:0/3145754949 conn(0x7ff2540d1f00 msgr2=0x7ff2540d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 -- 172.21.15.90:0/3145754949 shutdown_connections 2024-03-31T15:56:52.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.131+0000 7ff25bc00640 1 -- 172.21.15.90:0/3145754949 wait complete. 2024-03-31T15:56:52.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 Processor -- start 2024-03-31T15:56:52.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- start start 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff25409af80 0x7ff2540b58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 0x7ff25415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff254070670 con 0x7ff2540b15f0 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2540d12d0 con 0x7ff2540d2a20 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff25416b3c0 con 0x7ff25409af80 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41770/0 (socket says 172.21.15.90:41770) 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 -- 172.21.15.90:0/2201210727 learned_addr learned my addr 172.21.15.90:0/2201210727 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff259c00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 0x7ff25415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff259200640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff25409af80 0x7ff2540b58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 -- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff25409af80 msgr2=0x7ff2540b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff25409af80 0x7ff2540b58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 -- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 msgr2=0x7ff25415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 0x7ff25415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 -- 172.21.15.90:0/2201210727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25415de20 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff259c00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 0x7ff25415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff24be00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e870 tx=0x7ff23c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c017070 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23c00c940 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c00ccf0 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2540b4df0 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.135+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25415ed30 con 0x7ff2540b15f0 2024-03-31T15:56:52.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.139+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff254068d00 con 0x7ff2540b15f0 2024-03-31T15:56:52.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.139+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff23c01a070 con 0x7ff2540b15f0 2024-03-31T15:56:52.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.139+0000 7ff251600640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 0x7ff23006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.139+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7ff23c086940 con 0x7ff2540b15f0 2024-03-31T15:56:52.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.139+0000 7ff259200640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 0x7ff23006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.135 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.143+0000 7ff259200640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 0x7ff23006a870 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff2440ad000 tx=0x7ff244008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.143+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff23c056e00 con 0x7ff2540b15f0 2024-03-31T15:56:52.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.303+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff2540ce3c0 con 0x7ff2540b15f0 2024-03-31T15:56:52.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff251600640 1 -- 172.21.15.90:0/2201210727 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7ff23c056700 con 0x7ff2540b15f0 2024-03-31T15:56:52.300 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:52.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 msgr2=0x7ff23006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 0x7ff23006a870 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff2440ad000 tx=0x7ff244008040 comp rx=0 tx=0).stop 2024-03-31T15:56:52.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 msgr2=0x7ff254154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00e870 tx=0x7ff23c00ec70 comp rx=0 tx=0).stop 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 shutdown_connections 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff2300683b0 0x7ff23006a870 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2540d2a20 0x7ff25415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2540b15f0 0x7ff254154320 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 --2- 172.21.15.90:0/2201210727 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff25409af80 0x7ff2540b58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 >> 172.21.15.90:0/2201210727 conn(0x7ff2540d1f00 msgr2=0x7ff2540aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 shutdown_connections 2024-03-31T15:56:52.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.307+0000 7ff25bc00640 1 -- 172.21.15.90:0/2201210727 wait complete. 2024-03-31T15:56:52.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:52.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 Processor -- start 2024-03-31T15:56:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 -- start start 2024-03-31T15:56:52.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f929815c320 0x7f92980cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92980cd900 con 0x7f92980cd470 2024-03-31T15:56:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f92980aed70 con 0x7f929815c320 2024-03-31T15:56:52.471 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ea00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f92980cefc0 con 0x7f92980d2b20 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f9297400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f9297400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50672/0 (socket says 172.21.15.90:50672) 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f9297e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f929815c320 0x7f92980cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f9297400640 1 -- 172.21.15.90:0/2307749072 learned_addr learned my addr 172.21.15.90:0/2307749072 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.475+0000 7f929ca00640 1 --1- 172.21.15.90:0/2307749072 >> v1:172.21.15.102:6789/0 conn(0x7f92980cd470 0x7f92980cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:54348/0 (socket says 172.21.15.90:54348) 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9297400640 1 -- 172.21.15.90:0/2307749072 >> v2:172.21.15.102:3300/0 conn(0x7f929815c320 msgr2=0x7f92980cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9297400640 1 --2- 172.21.15.90:0/2307749072 >> v2:172.21.15.102:3300/0 conn(0x7f929815c320 0x7f92980cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9297400640 1 -- 172.21.15.90:0/2307749072 >> v1:172.21.15.102:6789/0 conn(0x7f92980cd470 legacy=0x7f92980cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9297400640 1 -- 172.21.15.90:0/2307749072 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92980cf2a0 con 0x7f92980d2b20 2024-03-31T15:56:52.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9297400640 1 --2- 172.21.15.90:0/2307749072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9284009870 tx=0x7f928401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b033eccf8147236 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9296a00640 1 -- 172.21.15.90:0/2307749072 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92840b2070 con 0x7f92980d2b20 2024-03-31T15:56:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9296a00640 1 -- 172.21.15.90:0/2307749072 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92840ad680 con 0x7f92980d2b20 2024-03-31T15:56:52.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f9296a00640 1 -- 172.21.15.90:0/2307749072 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92840ada50 con 0x7f92980d2b20 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 -- 172.21.15.90:0/2307749072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 msgr2=0x7f92980ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 --2- 172.21.15.90:0/2307749072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9284009870 tx=0x7f928401c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 -- 172.21.15.90:0/2307749072 shutdown_connections 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 --2- 172.21.15.90:0/2307749072 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980d2b20 0x7f92980ccf30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 --2- 172.21.15.90:0/2307749072 >> v2:172.21.15.102:3300/0 conn(0x7f929815c320 0x7f92980cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 -- 172.21.15.90:0/2307749072 >> 172.21.15.90:0/2307749072 conn(0x7f92980d1f20 msgr2=0x7f92980d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.479+0000 7f929ea00640 1 -- 172.21.15.90:0/2307749072 shutdown_connections 2024-03-31T15:56:52.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 -- 172.21.15.90:0/2307749072 wait complete. 2024-03-31T15:56:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 Processor -- start 2024-03-31T15:56:52.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 -- start start 2024-03-31T15:56:52.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:50682/0 (socket says 172.21.15.90:50682) 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980d2b20 0x7f92980b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 -- 172.21.15.90:0/1679741078 learned_addr learned my addr 172.21.15.90:0/1679741078 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f929815c320 0x7f9298156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9298168ec0 con 0x7f929815c320 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f92980d0eb0 con 0x7f92980d2b20 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f9298169040 con 0x7f92980cd470 2024-03-31T15:56:52.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f929ca00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f929815c320 0x7f9298156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297400640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980d2b20 0x7f92980b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 -- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980d2b20 msgr2=0x7f92980b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980d2b20 0x7f92980b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 -- 172.21.15.90:0/1679741078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f929815c320 msgr2=0x7f9298156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f929815c320 0x7f9298156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.483+0000 7f9297e00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929815d940 con 0x7f92980cd470 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f9297e00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f928800e870 tx=0x7f928800ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288017070 con 0x7f92980cd470 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928800c940 con 0x7f92980cd470 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929815e340 con 0x7f92980cd470 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928800ccf0 con 0x7f92980cd470 2024-03-31T15:56:52.481 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f929815e900 con 0x7f92980cd470 2024-03-31T15:56:52.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92980685f0 con 0x7f92980cd470 2024-03-31T15:56:52.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f928801a070 con 0x7f92980cd470 2024-03-31T15:56:52.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 0x7f927806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f9288086d20 con 0x7f92980cd470 2024-03-31T15:56:52.483 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.487+0000 7f9297400640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 0x7f927806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.484 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.491+0000 7f9297400640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 0x7f927806a840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f92840af000 tx=0x7f92840a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.491+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f92880571e0 con 0x7f92980cd470 2024-03-31T15:56:52.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.647+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92980cbf20 con 0x7f92980cd470 2024-03-31T15:56:52.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f928fe00640 1 -- 172.21.15.90:0/1679741078 <== mon.2 v2:172.21.15.102:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7f9288056ae0 con 0x7f92980cd470 2024-03-31T15:56:52.644 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:52.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 msgr2=0x7f927806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 0x7f927806a840 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f92840af000 tx=0x7f92840a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:52.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 msgr2=0x7f929815f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f928800e870 tx=0x7f928800ec70 comp rx=0 tx=0).stop 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 shutdown_connections 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f9278068380 0x7f927806a840 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f929815c320 0x7f9298156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980d2b20 0x7f92980b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 --2- 172.21.15.90:0/1679741078 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f92980cd470 0x7f929815f210 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 >> 172.21.15.90:0/1679741078 conn(0x7f92980d1f20 msgr2=0x7f92980cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 shutdown_connections 2024-03-31T15:56:52.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.651+0000 7f929ea00640 1 -- 172.21.15.90:0/1679741078 wait complete. 2024-03-31T15:56:52.665 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.807+0000 7f592dc00640 1 Processor -- start 2024-03-31T15:56:52.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f592dc00640 1 -- start start 2024-03-31T15:56:52.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f592dc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 0x7f592806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f592dc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f592815e8c0 con 0x7f592806ce50 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f592dc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59280d0ff0 con 0x7f59280cdf20 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f592dc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f59280d12d0 con 0x7f592815e4e0 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f592806ce50 0x7f59280ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:60760/0 (socket says 172.21.15.90:60760) 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926000640 1 -- 172.21.15.90:0/4050017688 learned_addr learned my addr 172.21.15.90:0/4050017688 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926a00640 1 --2- 172.21.15.90:0/4050017688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 0x7f592806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5925600640 1 -- 172.21.15.90:0/4050017688 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588169240 0 0) 0x7f592815e8c0 con 0x7f592806ce50 2024-03-31T15:56:52.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5925600640 1 -- 172.21.15.90:0/4050017688 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f590c003530 con 0x7f592806ce50 2024-03-31T15:56:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926a00640 1 -- 172.21.15.90:0/4050017688 >> v1:172.21.15.102:6789/0 conn(0x7f59280cdf20 legacy=0x7f59280ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926a00640 1 -- 172.21.15.90:0/4050017688 >> v1:172.21.15.90:6789/0 conn(0x7f592806ce50 legacy=0x7f59280ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.811+0000 7f5926a00640 1 -- 172.21.15.90:0/4050017688 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59280cc710 con 0x7f592815e4e0 2024-03-31T15:56:52.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f5926a00640 1 --2- 172.21.15.90:0/4050017688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 0x7f592806c910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f59180097f0 tx=0x7f591801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd61689316ae8496 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f5925600640 1 -- 172.21.15.90:0/4050017688 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59180b0070 con 0x7f592815e4e0 2024-03-31T15:56:52.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f5925600640 1 -- 172.21.15.90:0/4050017688 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59180ab600 con 0x7f592815e4e0 2024-03-31T15:56:52.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f5925600640 1 -- 172.21.15.90:0/4050017688 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59180ab9d0 con 0x7f592815e4e0 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- 172.21.15.90:0/4050017688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 msgr2=0x7f592806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 --2- 172.21.15.90:0/4050017688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 0x7f592806c910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f59180097f0 tx=0x7f591801c450 comp rx=0 tx=0).stop 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- 172.21.15.90:0/4050017688 shutdown_connections 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 --2- 172.21.15.90:0/4050017688 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592815e4e0 0x7f592806c910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- 172.21.15.90:0/4050017688 >> 172.21.15.90:0/4050017688 conn(0x7f59280d1f20 msgr2=0x7f59280d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- 172.21.15.90:0/4050017688 shutdown_connections 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- 172.21.15.90:0/4050017688 wait complete. 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 Processor -- start 2024-03-31T15:56:52.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.815+0000 7f592dc00640 1 -- start start 2024-03-31T15:56:52.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592806ce50 0x7f5928149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f592815e4e0 0x7f59280caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f59280d0d90 con 0x7f59280cdf20 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f59280d0c10 con 0x7f592815e4e0 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f592816c850 con 0x7f592806ce50 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41784/0 (socket says 172.21.15.90:41784) 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 -- 172.21.15.90:0/1557280006 learned_addr learned my addr 172.21.15.90:0/1557280006 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926a00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592806ce50 0x7f5928149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5927400640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f592815e4e0 0x7f59280caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 -- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592806ce50 msgr2=0x7f5928149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592806ce50 0x7f5928149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 -- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f592815e4e0 msgr2=0x7f59280caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f592815e4e0 0x7f59280caf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 -- 172.21.15.90:0/1557280006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59281540d0 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f5926000640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f591400c420 tx=0x7f591400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914017070 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5914005cf0 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59281543b0 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59140072d0 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592814c280 con 0x7f59280cdf20 2024-03-31T15:56:52.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.819+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59280685f0 con 0x7f59280cdf20 2024-03-31T15:56:52.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.823+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f591401a070 con 0x7f59280cdf20 2024-03-31T15:56:52.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.823+0000 7f591ea00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 0x7f590806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:52.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.823+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f5914085dd0 con 0x7f59280cdf20 2024-03-31T15:56:52.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.823+0000 7f5926a00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 0x7f590806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:52.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.823+0000 7f5926a00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 0x7f590806a870 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59180ad000 tx=0x7f59180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:52.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.827+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5914055c40 con 0x7f59280cdf20 2024-03-31T15:56:52.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.983+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5928156800 con 0x7f59280cdf20 2024-03-31T15:56:52.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.983+0000 7f591ea00640 1 -- 172.21.15.90:0/1557280006 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7f5914087020 con 0x7f59280cdf20 2024-03-31T15:56:52.978 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 msgr2=0x7f590806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 0x7f590806a870 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f59180ad000 tx=0x7f59180a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 msgr2=0x7f59281536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f591400c420 tx=0x7f591400c8e0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 shutdown_connections 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f59080683b0 0x7f590806a870 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f592815e4e0 0x7f59280caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59280cdf20 0x7f59281536a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 --2- 172.21.15.90:0/1557280006 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f592806ce50 0x7f5928149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 >> 172.21.15.90:0/1557280006 conn(0x7f59280d1f20 msgr2=0x7f59280d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 shutdown_connections 2024-03-31T15:56:52.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:52.987+0000 7f592dc00640 1 -- 172.21.15.90:0/1557280006 wait complete. 2024-03-31T15:56:53.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-31T15:56:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 Processor -- start 2024-03-31T15:56:53.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 -- start start 2024-03-31T15:56:53.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac0ce910 0x7f18ac0cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f18ac15d240 0x7f18ac0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18ac0cfe90 0x7f18ac0d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f18ac0cf230 con 0x7f18ac0cfe90 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f18ac0cd460 con 0x7f18ac15d240 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18b1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f18ac0cd740 con 0x7f18ac0ce910 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18ac0cfe90 0x7f18ac0d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f18ac0cfe90 0x7f18ac0d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43286/0 (socket says 172.21.15.90:43286) 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 -- 172.21.15.90:0/3403937895 learned_addr learned my addr 172.21.15.90:0/3403937895 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18a9e00640 1 --2- 172.21.15.90:0/3403937895 >> v2:172.21.15.102:3300/0 conn(0x7f18ac15d240 0x7f18ac0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18aa800640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac0ce910 0x7f18ac0cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 -- 172.21.15.90:0/3403937895 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac0ce910 msgr2=0x7f18ac0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac0ce910 0x7f18ac0cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.151 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 -- 172.21.15.90:0/3403937895 >> v2:172.21.15.102:3300/0 conn(0x7f18ac15d240 msgr2=0x7f18ac0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 --2- 172.21.15.90:0/3403937895 >> v2:172.21.15.102:3300/0 conn(0x7f18ac15d240 0x7f18ac0ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.155+0000 7f18ab200640 1 -- 172.21.15.90:0/3403937895 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac0cda20 con 0x7f18ac0cfe90 2024-03-31T15:56:53.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18ab200640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0cfe90 0x7f18ac0d0320 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f18ac071e20 tx=0x7f18a401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b4f6a16c8662a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18a9400640 1 -- 172.21.15.90:0/3403937895 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a40a5640 con 0x7f18ac0cfe90 2024-03-31T15:56:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18a9400640 1 -- 172.21.15.90:0/3403937895 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a40a57e0 con 0x7f18ac0cfe90 2024-03-31T15:56:53.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18a9400640 1 -- 172.21.15.90:0/3403937895 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a40a5bb0 con 0x7f18ac0cfe90 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 -- 172.21.15.90:0/3403937895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0cfe90 msgr2=0x7f18ac0d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0cfe90 0x7f18ac0d0320 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f18ac071e20 tx=0x7f18a401c900 comp rx=0 tx=0).stop 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 -- 172.21.15.90:0/3403937895 shutdown_connections 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0cfe90 0x7f18ac0d0320 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 --2- 172.21.15.90:0/3403937895 >> v2:172.21.15.102:3300/0 conn(0x7f18ac15d240 0x7f18ac0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 --2- 172.21.15.90:0/3403937895 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac0ce910 0x7f18ac0cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 -- 172.21.15.90:0/3403937895 >> 172.21.15.90:0/3403937895 conn(0x7f18ac08bfd0 msgr2=0x7f18ac0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 -- 172.21.15.90:0/3403937895 shutdown_connections 2024-03-31T15:56:53.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.159+0000 7f18b1400640 1 -- 172.21.15.90:0/3403937895 wait complete. 2024-03-31T15:56:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 Processor -- start 2024-03-31T15:56:53.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 -- start start 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18ac0cfe90 0x7f18ac1495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac15d240 0x7f18ac151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac167d70 con 0x7f18ac0ce910 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac167bf0 con 0x7f18ac0cfe90 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18b1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f18ac167ef0 con 0x7f18ac15d240 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18aa800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18aa800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43294/0 (socket says 172.21.15.90:43294) 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18aa800640 1 -- 172.21.15.90:0/3471497977 learned_addr learned my addr 172.21.15.90:0/3471497977 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.163+0000 7f18a9e00640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18ac0cfe90 0x7f18ac1495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18ab200640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac15d240 0x7f18ac151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 -- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac15d240 msgr2=0x7f18ac151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac15d240 0x7f18ac151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 -- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18ac0cfe90 msgr2=0x7f18ac1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18ac0cfe90 0x7f18ac1495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 -- 172.21.15.90:0/3471497977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac0c79e0 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18aa800640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f189c00e870 tx=0x7f189c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c017070 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189c00c940 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c00ccf0 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ac0c7cc0 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ac15aed0 con 0x7f18ac0ce910 2024-03-31T15:56:53.163 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18ac068660 con 0x7f18ac0ce910 2024-03-31T15:56:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.167+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f189c01a070 con 0x7f18ac0ce910 2024-03-31T15:56:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.171+0000 7f18a2a00640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 0x7f188c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.171+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f189c086d00 con 0x7f18ac0ce910 2024-03-31T15:56:53.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.171+0000 7f18a9e00640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 0x7f188c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.171+0000 7f18a9e00640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 0x7f188c06a840 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18ac155470 tx=0x7f1898009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.175+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f189c056ae0 con 0x7f18ac0ce910 2024-03-31T15:56:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f18ac0ce6f0 con 0x7f18ac0ce910 2024-03-31T15:56:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18a2a00640 1 -- 172.21.15.90:0/3471497977 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7f189c05ade0 con 0x7f18ac0ce910 2024-03-31T15:56:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:listed 0 entries 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 msgr2=0x7f188c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 0x7f188c06a840 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f18ac155470 tx=0x7f1898009190 comp rx=0 tx=0).stop 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 msgr2=0x7f18ac15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f189c00e870 tx=0x7f189c00ec70 comp rx=0 tx=0).stop 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 shutdown_connections 2024-03-31T15:56:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f188c068380 0x7f188c06a840 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f18ac15d240 0x7f18ac151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18ac0cfe90 0x7f18ac1495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 --2- 172.21.15.90:0/3471497977 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18ac0ce910 0x7f18ac15b6d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 >> 172.21.15.90:0/3471497977 conn(0x7f18ac08bfd0 msgr2=0x7f18ac06fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 shutdown_connections 2024-03-31T15:56:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.343+0000 7f18b1400640 1 -- 172.21.15.90:0/3471497977 wait complete. 2024-03-31T15:56:53.358 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-03-31T15:56:53.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:53.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-31T15:56:53.376 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[42432]: fuse finished with error 0 and tester_r 0 2024-03-31T15:56:53.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:53.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:53.379 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-31T15:56:53.454 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:53.455 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:53.455 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:53.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:53.464 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:56:53.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:53.464 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-31T15:56:53.516 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-31T15:56:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:53.517 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:53.517 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:56:53.518 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:53.526 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:56:53.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:56:53.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:53.526 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-31T15:56:53.575 INFO:teuthology.orchestra.run.smithi090.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-31T15:56:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:53.576 INFO:tasks.cephfs.mount:Terminating background process 2024-03-31T15:56:53.576 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:56:53.576 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:56:53.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:56:53.585 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:56:53.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:56:53.585 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-31T15:56:53.636 INFO:teuthology.orchestra.run.smithi102.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-31T15:56:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:56:53.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:53.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 Processor -- start 2024-03-31T15:56:53.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 -- start start 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 0x7fe1d0162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d015f780 con 0x7fe1d006d7c0 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1d015f1c0 con 0x7fe1d006c7b0 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1d5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d01622b0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1cf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe1d006d7c0 0x7fe1d015e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57246/0 (socket says 172.21.15.90:57246) 2024-03-31T15:56:53.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1cf400640 1 -- 172.21.15.90:0/2725298576 learned_addr learned my addr 172.21.15.90:0/2725298576 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.779+0000 7fe1cea00640 1 --2- 172.21.15.90:0/2725298576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 0x7fe1d0162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113265013 0 0) 0x7fe1d015f780 con 0x7fe1d006d7c0 2024-03-31T15:56:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1b4003530 con 0x7fe1d006d7c0 2024-03-31T15:56:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1846243671 0 0) 0x7fe1d015f1c0 con 0x7fe1d006c7b0 2024-03-31T15:56:53.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1d015f780 con 0x7fe1d006c7b0 2024-03-31T15:56:53.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4020126720 0 0) 0x7fe1b4003530 con 0x7fe1d006d7c0 2024-03-31T15:56:53.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cea00640 1 -- 172.21.15.90:0/2725298576 >> v1:172.21.15.102:6789/0 conn(0x7fe1d006c7b0 legacy=0x7fe1d006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cea00640 1 -- 172.21.15.90:0/2725298576 >> v1:172.21.15.90:6789/0 conn(0x7fe1d006d7c0 legacy=0x7fe1d015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.777 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cea00640 1 -- 172.21.15.90:0/2725298576 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d015f4a0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cea00640 1 --2- 172.21.15.90:0/2725298576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 0x7fe1d0162740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00097f0 tx=0x7fe1c001c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfab4a054e6a7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00b0070 con 0x7fe1d015e1b0 2024-03-31T15:56:53.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c00ab400 con 0x7fe1d015e1b0 2024-03-31T15:56:53.779 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.783+0000 7fe1cd600640 1 -- 172.21.15.90:0/2725298576 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8001a00 con 0x7fe1d015e1b0 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- 172.21.15.90:0/2725298576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 msgr2=0x7fe1d0162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/2725298576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 0x7fe1d0162740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00097f0 tx=0x7fe1c001c340 comp rx=0 tx=0).stop 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- 172.21.15.90:0/2725298576 shutdown_connections 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/2725298576 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d015e1b0 0x7fe1d0162740 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- 172.21.15.90:0/2725298576 >> 172.21.15.90:0/2725298576 conn(0x7fe1d009aa40 msgr2=0x7fe1d009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- 172.21.15.90:0/2725298576 shutdown_connections 2024-03-31T15:56:53.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- 172.21.15.90:0/2725298576 wait complete. 2024-03-31T15:56:53.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 Processor -- start 2024-03-31T15:56:53.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.787+0000 7fe1d5c00640 1 -- start start 2024-03-31T15:56:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1d006c7b0 0x7fe1d01506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d006d7c0 0x7fe1d006f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00d2b50 con 0x7fe1d015e1b0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d00d29d0 con 0x7fe1d006c7b0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe1d00ce7a0 con 0x7fe1d006d7c0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43302/0 (socket says 172.21.15.90:43302) 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 -- 172.21.15.90:0/844614401 learned_addr learned my addr 172.21.15.90:0/844614401 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cea00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1d006c7b0 0x7fe1d01506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1ce000640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d006d7c0 0x7fe1d006f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 -- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d006d7c0 msgr2=0x7fe1d006f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d006d7c0 0x7fe1d006f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 -- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1d006c7b0 msgr2=0x7fe1d01506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:53.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1d006c7b0 0x7fe1d01506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 -- 172.21.15.90:0/844614401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d00ccfc0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1cf400640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8001360 tx=0x7fe1c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8017070 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1c8012400 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d00cd2a0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80127b0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d0157ab0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.791+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d00686d0 con 0x7fe1d015e1b0 2024-03-31T15:56:53.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.795+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe1c801a070 con 0x7fe1d015e1b0 2024-03-31T15:56:53.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.795+0000 7fe1c6a00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 0x7fe1b006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:53.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.795+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fe1c8086940 con 0x7fe1d015e1b0 2024-03-31T15:56:53.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.795+0000 7fe1cea00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 0x7fe1b006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:53.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.795+0000 7fe1cea00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 0x7fe1b006a840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00ad000 tx=0x7fe1c00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:53.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:53.799+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe1c8056710 con 0x7fe1d015e1b0 2024-03-31T15:56:54.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.119+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1d00b2130 con 0x7fe1d015e1b0 2024-03-31T15:56:54.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1c6a00640 1 -- 172.21.15.90:0/844614401 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7fe1c805aa10 con 0x7fe1d015e1b0 2024-03-31T15:56:54.116 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:54.116 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:54.116 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 msgr2=0x7fe1b006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 0x7fe1b006a840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe1c00ad000 tx=0x7fe1c00a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 msgr2=0x7fe1d006fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8001360 tx=0x7fe1c800a430 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 shutdown_connections 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe1b0068380 0x7fe1b006a840 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1d015e1b0 0x7fe1d006fa10 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe1d006d7c0 0x7fe1d006f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 --2- 172.21.15.90:0/844614401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1d006c7b0 0x7fe1d01506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 >> 172.21.15.90:0/844614401 conn(0x7fe1d009aa40 msgr2=0x7fe1d0160840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:54.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 shutdown_connections 2024-03-31T15:56:54.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.123+0000 7fe1d5c00640 1 -- 172.21.15.90:0/844614401 wait complete. 2024-03-31T15:56:54.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:54.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.279+0000 7fe9ee800640 1 Processor -- start 2024-03-31T15:56:54.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.279+0000 7fe9ee800640 1 -- start start 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9ee800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9e80ccdb0 0x7fe9e80b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9ee800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e80b34f0 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9ee800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e80b37d0 con 0x7fe9e806b820 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9ee800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e80b3ab0 con 0x7fe9e80ccdb0 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe9e80cc9d0 0x7fe9e8142030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57250/0 (socket says 172.21.15.90:57250) 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e7400640 1 -- 172.21.15.90:0/3849577616 learned_addr learned my addr 172.21.15.90:0/3849577616 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/3849577616 >> v2:172.21.15.102:3300/0 conn(0x7fe9e80ccdb0 0x7fe9e80b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.279 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25839153 0 0) 0x7fe9e80b34f0 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9cc003530 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104059361 0 0) 0x7fe9e80b37d0 con 0x7fe9e806b820 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e80b34f0 con 0x7fe9e806b820 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1397857910 0 0) 0x7fe9cc003530 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 >> v2:172.21.15.102:3300/0 conn(0x7fe9e80ccdb0 msgr2=0x7fe9e80b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 --2- 172.21.15.90:0/3849577616 >> v2:172.21.15.102:3300/0 conn(0x7fe9e80ccdb0 0x7fe9e80b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 >> v1:172.21.15.102:6789/0 conn(0x7fe9e806b820 legacy=0x7fe9e80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e8141830 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fe9d8001680 con 0x7fe9e80cc9d0 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 >> v1:172.21.15.90:6789/0 conn(0x7fe9e80cc9d0 legacy=0x7fe9e8142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc003ba0 0x7fe9cc003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 0x7fe9cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc005220 0x7fe9cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80b3ab0 con 0x7fe9cc004810 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc00c150 con 0x7fe9cc003ba0 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc00c350 con 0x7fe9cc005220 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 0x7fe9cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e7400640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc003ba0 0x7fe9cc003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e7e00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc005220 0x7fe9cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 -- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc005220 msgr2=0x7fe9cc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc005220 0x7fe9cc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 -- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc003ba0 msgr2=0x7fe9cc003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc003ba0 0x7fe9cc003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9dfe00640 1 -- 172.21.15.90:0/3849577616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc003530 con 0x7fe9cc004810 2024-03-31T15:56:54.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.283+0000 7fe9e7e00640 1 -- 172.21.15.90:0/3849577616 reap_dead start 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 0x7fe9cc004c90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009ae0 tx=0x7fe9d001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e703da2bbfc9afe server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00b2070 con 0x7fe9cc004810 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d00a5600 con 0x7fe9cc004810 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9e6a00640 1 -- 172.21.15.90:0/3849577616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d00096b0 con 0x7fe9cc004810 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9ee800640 1 -- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 msgr2=0x7fe9cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.287+0000 7fe9ee800640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 0x7fe9cc004c90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009ae0 tx=0x7fe9d001ca90 comp rx=0 tx=0).stop 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- 172.21.15.90:0/3849577616 shutdown_connections 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 --2- 172.21.15.90:0/3849577616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9cc004810 0x7fe9cc004c90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- 172.21.15.90:0/3849577616 >> 172.21.15.90:0/3849577616 conn(0x7fe9e80b1c80 msgr2=0x7fe9e80b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- 172.21.15.90:0/3849577616 shutdown_connections 2024-03-31T15:56:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- 172.21.15.90:0/3849577616 wait complete. 2024-03-31T15:56:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 Processor -- start 2024-03-31T15:56:54.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- start start 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc004810 0x7fe9e81536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 0x7fe9e8140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80cef70 con 0x7fe9e8155800 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e80cedf0 con 0x7fe9e81553b0 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e80cf0f0 con 0x7fe9cc004810 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43320/0 (socket says 172.21.15.90:43320) 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 -- 172.21.15.90:0/2880328022 learned_addr learned my addr 172.21.15.90:0/2880328022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 0x7fe9e8140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7400640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc004810 0x7fe9e81536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 -- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc004810 msgr2=0x7fe9e81536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc004810 0x7fe9e81536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 -- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 msgr2=0x7fe9e8140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 0x7fe9e8140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 -- 172.21.15.90:0/2880328022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e8155f00 con 0x7fe9e8155800 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9dfe00640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 0x7fe9e8140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:54.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9e7e00640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806c430 tx=0x7fe9e000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0019070 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e0006e30 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0003b80 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.291+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e81561a0 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e813ff10 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e80686d0 con 0x7fe9e8155800 2024-03-31T15:56:54.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe9e000d7a0 con 0x7fe9e8155800 2024-03-31T15:56:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9df400640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 0x7fe9c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fe9e0085ef0 con 0x7fe9e8155800 2024-03-31T15:56:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.295+0000 7fe9e7400640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 0x7fe9c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.299+0000 7fe9e7400640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 0x7fe9c406a840 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c10 tx=0x7fe9d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.299+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe9e0056280 con 0x7fe9e8155800 2024-03-31T15:56:54.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.483+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9e80b0f20 con 0x7fe9e8155800 2024-03-31T15:56:54.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9df400640 1 -- 172.21.15.90:0/2880328022 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7fe9e0055b80 con 0x7fe9e8155800 2024-03-31T15:56:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:54.480 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 msgr2=0x7fe9c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 0x7fe9c406a840 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c10 tx=0x7fe9d8009040 comp rx=0 tx=0).stop 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 msgr2=0x7fe9e8140970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806c430 tx=0x7fe9e000ff70 comp rx=0 tx=0).stop 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 shutdown_connections 2024-03-31T15:56:54.486 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9c4068380 0x7fe9c406a840 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9e8155800 0x7fe9e8140970 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9e81553b0 0x7fe9e8140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 --2- 172.21.15.90:0/2880328022 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9cc004810 0x7fe9e81536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 >> 172.21.15.90:0/2880328022 conn(0x7fe9e80b1c80 msgr2=0x7fe9e813d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 shutdown_connections 2024-03-31T15:56:54.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.487+0000 7fe9ee800640 1 -- 172.21.15.90:0/2880328022 wait complete. 2024-03-31T15:56:54.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-31T15:53:35.236139+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-31T15:56:24.733192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-31T15:56:25.363340+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-31T15:56:54.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-31T15:56:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.647+0000 7fc19a600640 1 Processor -- start 2024-03-31T15:56:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 -- start start 2024-03-31T15:56:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc19415b040 0x7fc19415b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc194071940 0x7fc194071e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc194072a60 con 0x7fc1940713f0 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fc1940b4a60 con 0x7fc194071940 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc19a600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc194072d40 con 0x7fc19415b040 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc194071940 0x7fc194071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fc194071940 0x7fc194071e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43334/0 (socket says 172.21.15.90:43334) 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 -- 172.21.15.90:0/465443417 learned_addr learned my addr 172.21.15.90:0/465443417 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc193400640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc19415b040 0x7fc19415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 -- 172.21.15.90:0/465443417 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc19415b040 msgr2=0x7fc19415b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc19415b040 0x7fc19415b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 -- 172.21.15.90:0/465443417 >> v1:172.21.15.102:6789/0 conn(0x7fc1940713f0 legacy=0x7fc1940ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.647 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.651+0000 7fc192a00640 1 -- 172.21.15.90:0/465443417 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1940aeac0 con 0x7fc194071940 2024-03-31T15:56:54.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc192a00640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc194071940 0x7fc194071e90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb70ca62e64404e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc192000640 1 -- 172.21.15.90:0/465443417 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800a7070 con 0x7fc194071940 2024-03-31T15:56:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc192000640 1 -- 172.21.15.90:0/465443417 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1800b0070 con 0x7fc194071940 2024-03-31T15:56:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc192000640 1 -- 172.21.15.90:0/465443417 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800ab890 con 0x7fc194071940 2024-03-31T15:56:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 -- 172.21.15.90:0/465443417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc194071940 msgr2=0x7fc194071e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc194071940 0x7fc194071e90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18001c560 comp rx=0 tx=0).stop 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 -- 172.21.15.90:0/465443417 shutdown_connections 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc194071940 0x7fc194071e90 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 --2- 172.21.15.90:0/465443417 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc19415b040 0x7fc19415b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 -- 172.21.15.90:0/465443417 >> 172.21.15.90:0/465443417 conn(0x7fc1940b18b0 msgr2=0x7fc1940b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 -- 172.21.15.90:0/465443417 shutdown_connections 2024-03-31T15:56:54.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.655+0000 7fc19a600640 1 -- 172.21.15.90:0/465443417 wait complete. 2024-03-31T15:56:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 Processor -- start 2024-03-31T15:56:54.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- start start 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc194071940 0x7fc194154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19415b040 0x7fc1940af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc194167e30 con 0x7fc1940713f0 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940727e0 con 0x7fc19415b040 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fc194167fb0 con 0x7fc194071940 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43346/0 (socket says 172.21.15.90:43346) 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 -- 172.21.15.90:0/3242247163 learned_addr learned my addr 172.21.15.90:0/3242247163 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193e00640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19415b040 0x7fc1940af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc192a00640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc194071940 0x7fc194154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 -- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc194071940 msgr2=0x7fc194154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc194071940 0x7fc194154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 -- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19415b040 msgr2=0x7fc1940af370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19415b040 0x7fc1940af370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 -- 172.21.15.90:0/3242247163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1941551a0 con 0x7fc1940713f0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc193400640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc18400d620 tx=0x7fc18400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18400b620 con 0x7fc1940713f0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc184019070 con 0x7fc1940713f0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18400b9e0 con 0x7fc1940713f0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc194155440 con 0x7fc1940713f0 2024-03-31T15:56:54.655 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc194155a00 con 0x7fc1940713f0 2024-03-31T15:56:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.659+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc194075880 con 0x7fc1940713f0 2024-03-31T15:56:54.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.663+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc18400bb80 con 0x7fc1940713f0 2024-03-31T15:56:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.663+0000 7fc18b400640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 0x7fc17406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.663+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fc184086ff0 con 0x7fc1940713f0 2024-03-31T15:56:54.657 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.663+0000 7fc192a00640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 0x7fc17406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:54.658 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.663+0000 7fc192a00640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 0x7fc17406a840 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc1800af000 tx=0x7fc1800a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:54.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.667+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fc184057330 con 0x7fc1940713f0 2024-03-31T15:56:54.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc1940d0b50 con 0x7fc1940713f0 2024-03-31T15:56:54.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc18b400640 1 -- 172.21.15.90:0/3242247163 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) ==== 93+0+10 (secure 0 0 0) 0x7fc184056c30 con 0x7fc1940713f0 2024-03-31T15:56:54.869 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 msgr2=0x7fc17406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 0x7fc17406a840 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc1800af000 tx=0x7fc1800a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 msgr2=0x7fc1940b0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc18400d620 tx=0x7fc18400dae0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 shutdown_connections 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fc174068380 0x7fc17406a840 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc19415b040 0x7fc1940af370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fc194071940 0x7fc194154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 --2- 172.21.15.90:0/3242247163 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc1940713f0 0x7fc1940b0480 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 >> 172.21.15.90:0/3242247163 conn(0x7fc1940b18b0 msgr2=0x7fc194159650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 shutdown_connections 2024-03-31T15:56:54.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:54.875+0000 7fc19a600640 1 -- 172.21.15.90:0/3242247163 wait complete. 2024-03-31T15:56:54.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-31T15:56:55.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 Processor -- start 2024-03-31T15:56:55.029 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 -- start start 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee3406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee3406ce50 0x7fee3406d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 0x7fee340cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fee3415e8c0 con 0x7fee3406ce50 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fee340d0ff0 con 0x7fee340ce7a0 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee39000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fee340d12d0 con 0x7fee3415e4e0 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee3406ce50 0x7fee3406d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fee3406ce50 0x7fee3406d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43348/0 (socket says 172.21.15.90:43348) 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 -- 172.21.15.90:0/2393276421 learned_addr learned my addr 172.21.15.90:0/2393276421 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:55.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee32800640 1 --2- 172.21.15.90:0/2393276421 >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 0x7fee340cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee31e00640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee3406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 -- 172.21.15.90:0/2393276421 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 msgr2=0x7fee3406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee3406c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 -- 172.21.15.90:0/2393276421 >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 msgr2=0x7fee340cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 --2- 172.21.15.90:0/2393276421 >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 0x7fee340cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee2be00640 1 -- 172.21.15.90:0/2393276421 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee340cc710 con 0x7fee3406ce50 2024-03-31T15:56:55.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.035+0000 7fee32800640 1 --2- 172.21.15.90:0/2393276421 >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 0x7fee340cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:55.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee2be00640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee3406d290 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fee34075360 tx=0x7fee1c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9888c6c6156d7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee31400640 1 -- 172.21.15.90:0/2393276421 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0b0070 con 0x7fee3406ce50 2024-03-31T15:56:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee31400640 1 -- 172.21.15.90:0/2393276421 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c0ab600 con 0x7fee3406ce50 2024-03-31T15:56:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee31400640 1 -- 172.21.15.90:0/2393276421 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0ab9d0 con 0x7fee3406ce50 2024-03-31T15:56:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 -- 172.21.15.90:0/2393276421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 msgr2=0x7fee3406d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.034 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee3406d290 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fee34075360 tx=0x7fee1c01c490 comp rx=0 tx=0).stop 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 -- 172.21.15.90:0/2393276421 shutdown_connections 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 --2- 172.21.15.90:0/2393276421 >> v2:172.21.15.102:3300/0 conn(0x7fee340ce7a0 0x7fee340cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee3406d290 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 --2- 172.21.15.90:0/2393276421 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee3406c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 -- 172.21.15.90:0/2393276421 >> 172.21.15.90:0/2393276421 conn(0x7fee340d1f20 msgr2=0x7fee340d2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 -- 172.21.15.90:0/2393276421 shutdown_connections 2024-03-31T15:56:55.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.039+0000 7fee39000640 1 -- 172.21.15.90:0/2393276421 wait complete. 2024-03-31T15:56:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 Processor -- start 2024-03-31T15:56:55.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- start start 2024-03-31T15:56:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee340ce7a0 0x7fee34153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee341539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fee340cd090 con 0x7fee3406ce50 2024-03-31T15:56:55.038 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee340ccf10 con 0x7fee340ce7a0 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fee340cd210 con 0x7fee3415e4e0 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43358/0 (socket says 172.21.15.90:43358) 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 -- 172.21.15.90:0/4085037319 learned_addr learned my addr 172.21.15.90:0/4085037319 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee2be00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee340ce7a0 0x7fee34153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee32800640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee341539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 -- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 msgr2=0x7fee341539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee341539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 -- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee340ce7a0 msgr2=0x7fee34153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee340ce7a0 0x7fee34153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 -- 172.21.15.90:0/4085037319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3415cf80 con 0x7fee3406ce50 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee32800640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee341539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:55.039 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee31e00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fee2400e870 tx=0x7fee2400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee24017070 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2400c940 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2400ccf0 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3409cc30 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3409d1f0 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.043+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee340685f0 con 0x7fee3406ce50 2024-03-31T15:56:55.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.047+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fee2401a070 con 0x7fee3406ce50 2024-03-31T15:56:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.047+0000 7fee2a000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 0x7fee1006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.047+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fee24086d70 con 0x7fee3406ce50 2024-03-31T15:56:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.047+0000 7fee2be00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 0x7fee1006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.047+0000 7fee2be00640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 0x7fee1006a840 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0ad000 tx=0x7fee1c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.051+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fee24056ab0 con 0x7fee3406ce50 2024-03-31T15:56:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fee340c62a0 con 0x7fee3406ce50 2024-03-31T15:56:55.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee2a000640 1 -- 172.21.15.90:0/4085037319 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) ==== 104+0+11 (secure 0 0 0) 0x7fee2405adb0 con 0x7fee3406ce50 2024-03-31T15:56:55.205 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 msgr2=0x7fee1006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 0x7fee1006a840 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0ad000 tx=0x7fee1c0a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 msgr2=0x7fee34154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fee2400e870 tx=0x7fee2400ec70 comp rx=0 tx=0).stop 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 shutdown_connections 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fee10068380 0x7fee1006a840 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fee3415e4e0 0x7fee341539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee340ce7a0 0x7fee34153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 --2- 172.21.15.90:0/4085037319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fee3406ce50 0x7fee34154230 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.211+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 >> 172.21.15.90:0/4085037319 conn(0x7fee340d1f20 msgr2=0x7fee34156ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.215+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 shutdown_connections 2024-03-31T15:56:55.209 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.215+0000 7fee39000640 1 -- 172.21.15.90:0/4085037319 wait complete. 2024-03-31T15:56:55.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-31T15:56:55.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 Processor -- start 2024-03-31T15:56:55.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 -- start start 2024-03-31T15:56:55.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 0x7eff100709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff10161d80 con 0x7eff10162540 2024-03-31T15:56:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1015efc0 con 0x7eff100cb8c0 2024-03-31T15:56:55.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff16000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7eff101620b0 con 0x7eff100cbca0 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7eff10162540 0x7eff100b2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57260/0 (socket says 172.21.15.90:57260) 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0fe00640 1 -- 172.21.15.90:0/1621565819 learned_addr learned my addr 172.21.15.90:0/1621565819 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff07e00640 1 --2- 172.21.15.90:0/1621565819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 0x7eff100709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434608828 0 0) 0x7eff10161d80 con 0x7eff10162540 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef4003530 con 0x7eff10162540 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345943583 0 0) 0x7eff1015efc0 con 0x7eff100cb8c0 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff10161d80 con 0x7eff100cb8c0 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff07e00640 1 -- 172.21.15.90:0/1621565819 >> v1:172.21.15.102:6789/0 conn(0x7eff100cb8c0 legacy=0x7eff1015e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff07e00640 1 -- 172.21.15.90:0/1621565819 >> v1:172.21.15.90:6789/0 conn(0x7eff10162540 legacy=0x7eff100b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.379+0000 7eff07e00640 1 -- 172.21.15.90:0/1621565819 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1015f2a0 con 0x7eff100cbca0 2024-03-31T15:56:55.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff07e00640 1 --2- 172.21.15.90:0/1621565819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 0x7eff100709a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efef80097f0 tx=0x7efef801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6d969d97c174f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef80aa430 con 0x7eff100cbca0 2024-03-31T15:56:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef80af070 con 0x7eff100cbca0 2024-03-31T15:56:55.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff0ea00640 1 -- 172.21.15.90:0/1621565819 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef80aa7f0 con 0x7eff100cbca0 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 -- 172.21.15.90:0/1621565819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 msgr2=0x7eff100709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 --2- 172.21.15.90:0/1621565819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 0x7eff100709a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efef80097f0 tx=0x7efef801c4b0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 -- 172.21.15.90:0/1621565819 shutdown_connections 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 --2- 172.21.15.90:0/1621565819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cbca0 0x7eff100709a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 -- 172.21.15.90:0/1621565819 >> 172.21.15.90:0/1621565819 conn(0x7eff100cde90 msgr2=0x7eff100ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 -- 172.21.15.90:0/1621565819 shutdown_connections 2024-03-31T15:56:55.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.383+0000 7eff16000640 1 -- 172.21.15.90:0/1621565819 wait complete. 2024-03-31T15:56:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 Processor -- start 2024-03-31T15:56:55.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 -- start start 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cb8c0 0x7eff100ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 0x7eff101582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7eff1016c2b0 con 0x7eff10162540 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7eff1016c130 con 0x7eff100cbca0 2024-03-31T15:56:55.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff16000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7eff1016c430 con 0x7eff100cb8c0 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43374/0 (socket says 172.21.15.90:43374) 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 -- 172.21.15.90:0/1175690236 learned_addr learned my addr 172.21.15.90:0/1175690236 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0f400640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cb8c0 0x7eff100ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff07e00640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 0x7eff101582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 -- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cb8c0 msgr2=0x7eff100ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cb8c0 0x7eff100ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 -- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 msgr2=0x7eff101582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 0x7eff101582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff0fe00640 1 -- 172.21.15.90:0/1175690236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff100ca640 con 0x7eff10162540 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.387+0000 7eff07e00640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 0x7eff101582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:55.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff0fe00640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7eff0800cb50 tx=0x7eff0800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08017070 con 0x7eff10162540 2024-03-31T15:56:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff0800ad40 con 0x7eff10162540 2024-03-31T15:56:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff10158800 con 0x7eff10162540 2024-03-31T15:56:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08012620 con 0x7eff10162540 2024-03-31T15:56:55.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff100c8780 con 0x7eff10162540 2024-03-31T15:56:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff100685f0 con 0x7eff10162540 2024-03-31T15:56:55.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7eff0801a070 con 0x7eff10162540 2024-03-31T15:56:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.391+0000 7eff07400640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 0x7efeec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.395+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7eff080868a0 con 0x7eff10162540 2024-03-31T15:56:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.395+0000 7eff0f400640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 0x7efeec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.395+0000 7eff0f400640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 0x7efeec06a840 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7eff000015e0 tx=0x7eff00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.395+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7eff100ca640 con 0x7eff10162540 2024-03-31T15:56:55.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7eff10169f00 con 0x7eff10162540 2024-03-31T15:56:55.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff07400640 1 -- 172.21.15.90:0/1175690236 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) ==== 100+0+11 (secure 0 0 0) 0x7eff08056d60 con 0x7eff10162540 2024-03-31T15:56:55.549 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 64 2024-03-31T15:56:55.551 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 msgr2=0x7efeec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 0x7efeec06a840 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7eff000015e0 tx=0x7eff00009040 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 msgr2=0x7eff100aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7eff0800cb50 tx=0x7eff0800a430 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 shutdown_connections 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7efeec068380 0x7efeec06a840 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7eff10162540 0x7eff100aef00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff100cbca0 0x7eff101582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 --2- 172.21.15.90:0/1175690236 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7eff100cb8c0 0x7eff100ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 >> 172.21.15.90:0/1175690236 conn(0x7eff100cde90 msgr2=0x7eff1015f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 shutdown_connections 2024-03-31T15:56:55.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.555+0000 7eff16000640 1 -- 172.21.15.90:0/1175690236 wait complete. 2024-03-31T15:56:55.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:55.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 Processor -- start 2024-03-31T15:56:55.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 -- start start 2024-03-31T15:56:55.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5815c120 0x7f7d5815fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 0x7f7d580b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d580cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f7d58160bf0 con 0x7f7d5815c120 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7d580d2b10 con 0x7f7d580d2680 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d5e200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7d58160350 con 0x7f7d580b2c20 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d57400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5815c120 0x7f7d5815fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d57400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7d5815c120 0x7f7d5815fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43386/0 (socket says 172.21.15.90:43386) 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d57400640 1 -- 172.21.15.90:0/2296553819 learned_addr learned my addr 172.21.15.90:0/2296553819 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d57e00640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d580cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.711+0000 7f7d4fe00640 1 --2- 172.21.15.90:0/2296553819 >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 0x7f7d580b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 -- 172.21.15.90:0/2296553819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 msgr2=0x7f7d580cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d580cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 -- 172.21.15.90:0/2296553819 >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 msgr2=0x7f7d580b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:55.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 --2- 172.21.15.90:0/2296553819 >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 0x7f7d580b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 -- 172.21.15.90:0/2296553819 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d58160630 con 0x7f7d5815c120 2024-03-31T15:56:55.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d4fe00640 1 --2- 172.21.15.90:0/2296553819 >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 0x7f7d580b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d57400640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d5815fd00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7d5806ed50 tx=0x7f7d4801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893e5d1c95aed80f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d56a00640 1 -- 172.21.15.90:0/2296553819 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480b0070 con 0x7f7d5815c120 2024-03-31T15:56:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d56a00640 1 -- 172.21.15.90:0/2296553819 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d480ab600 con 0x7f7d5815c120 2024-03-31T15:56:55.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d56a00640 1 -- 172.21.15.90:0/2296553819 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480ab9d0 con 0x7f7d5815c120 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 -- 172.21.15.90:0/2296553819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 msgr2=0x7f7d5815fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d5815fd00 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7d5806ed50 tx=0x7f7d4801c3d0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 -- 172.21.15.90:0/2296553819 shutdown_connections 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d580cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 --2- 172.21.15.90:0/2296553819 >> v2:172.21.15.102:3300/0 conn(0x7f7d580d2680 0x7f7d580b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 --2- 172.21.15.90:0/2296553819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d5815fd00 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.715+0000 7f7d5e200640 1 -- 172.21.15.90:0/2296553819 >> 172.21.15.90:0/2296553819 conn(0x7f7d5809aa40 msgr2=0x7f7d5809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.719+0000 7f7d5e200640 1 -- 172.21.15.90:0/2296553819 shutdown_connections 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.719+0000 7f7d5e200640 1 -- 172.21.15.90:0/2296553819 wait complete. 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.719+0000 7f7d5e200640 1 Processor -- start 2024-03-31T15:56:55.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.719+0000 7f7d5e200640 1 -- start start 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d58148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 0x7f7d580af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d580cea90 con 0x7f7d5815c120 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d580ce910 con 0x7f7d580d2680 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f7d580cec10 con 0x7f7d580b2c20 2024-03-31T15:56:55.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43400/0 (socket says 172.21.15.90:43400) 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 -- 172.21.15.90:0/68870830 learned_addr learned my addr 172.21.15.90:0/68870830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d4fe00640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 0x7f7d580af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57400640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d58148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 -- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 msgr2=0x7f7d58148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d58148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 -- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 msgr2=0x7f7d580af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 0x7f7d580af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 -- 172.21.15.90:0/68870830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d58150950 con 0x7f7d5815c120 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d4fe00640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 0x7f7d580af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d57e00640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000eed0 tx=0x7f7d5000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50018020 con 0x7f7d5815c120 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d50014400 con 0x7f7d5815c120 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d500147f0 con 0x7f7d5815c120 2024-03-31T15:56:55.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d58150c30 con 0x7f7d5815c120 2024-03-31T15:56:55.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d581568a0 con 0x7f7d5815c120 2024-03-31T15:56:55.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.723+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d580686d0 con 0x7f7d5815c120 2024-03-31T15:56:55.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.727+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7d5001b020 con 0x7f7d5815c120 2024-03-31T15:56:55.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.727+0000 7f7d4f400640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 0x7f7d3406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:55.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.727+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f7d500871d0 con 0x7f7d5815c120 2024-03-31T15:56:55.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.727+0000 7f7d57400640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 0x7f7d3406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:55.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.727+0000 7f7d57400640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 0x7f7d3406a840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7d480ad000 tx=0x7f7d480a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.731+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7d50022070 con 0x7f7d5815c120 2024-03-31T15:56:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.919+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d580b1b60 con 0x7f7d5815c120 2024-03-31T15:56:55.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.919+0000 7f7d4f400640 1 -- 172.21.15.90:0/68870830 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7f7d50057560 con 0x7f7d5815c120 2024-03-31T15:56:55.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:55.915 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:55.915 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 msgr2=0x7f7d3406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 0x7f7d3406a840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7d480ad000 tx=0x7f7d480a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 msgr2=0x7f7d580afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000eed0 tx=0x7f7d5000b480 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 shutdown_connections 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f7d34068380 0x7f7d3406a840 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d5815c120 0x7f7d580afc30 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d580d2680 0x7f7d580af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 --2- 172.21.15.90:0/68870830 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7d580b2c20 0x7f7d58148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 >> 172.21.15.90:0/68870830 conn(0x7f7d5809aa40 msgr2=0x7f7d58158fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:55.918 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 shutdown_connections 2024-03-31T15:56:55.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:55.923+0000 7f7d5e200640 1 -- 172.21.15.90:0/68870830 wait complete. 2024-03-31T15:56:55.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:56.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.075+0000 7f8cd1400640 1 Processor -- start 2024-03-31T15:56:56.071 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.075+0000 7f8cd1400640 1 -- start start 2024-03-31T15:56:56.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc072ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd1400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ccc15f9a0 0x7f8ccc0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd1400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ccc1603f0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd1400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8ccc1606d0 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8ccc0722d0 con 0x7f8ccc145740 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cc3e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8ccc15f9a0 0x7f8ccc0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd0a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8ccc15fe20 0x7f8ccc0701b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57264/0 (socket says 172.21.15.90:57264) 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cd0a00640 1 -- 172.21.15.90:0/2183637973 learned_addr learned my addr 172.21.15.90:0/2183637973 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:56.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8ccbe00640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc072ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cc3e00640 1 -- 172.21.15.90:0/2183637973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 msgr2=0x7f8ccc072ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cc3e00640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc072ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cc3e00640 1 -- 172.21.15.90:0/2183637973 >> v1:172.21.15.90:6789/0 conn(0x7f8ccc15fe20 legacy=0x7f8ccc0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8cc3e00640 1 -- 172.21.15.90:0/2183637973 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc0725b0 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.079+0000 7f8ccb400640 1 -- 172.21.15.90:0/2183637973 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362071434 0 0) 0x7f8ccc1603f0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cc3e00640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0cb1a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40097c0 tx=0x7f8cb401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7277b71ad94f7c77 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8ccb400640 1 -- 172.21.15.90:0/2183637973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40b0070 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8ccb400640 1 -- 172.21.15.90:0/2183637973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb40ab600 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8ccb400640 1 -- 172.21.15.90:0/2183637973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40ab9d0 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- 172.21.15.90:0/2183637973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15f9a0 msgr2=0x7f8ccc0cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0cb1a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40097c0 tx=0x7f8cb401c4a0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- 172.21.15.90:0/2183637973 shutdown_connections 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0cb1a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 --2- 172.21.15.90:0/2183637973 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc072ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.077 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- 172.21.15.90:0/2183637973 >> 172.21.15.90:0/2183637973 conn(0x7f8ccc0cd660 msgr2=0x7f8ccc0cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- 172.21.15.90:0/2183637973 shutdown_connections 2024-03-31T15:56:56.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- 172.21.15.90:0/2183637973 wait complete. 2024-03-31T15:56:56.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 Processor -- start 2024-03-31T15:56:56.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.083+0000 7f8cd1400640 1 -- start start 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc1696d0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8ccc169550 con 0x7f8ccc15f9a0 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8ccc169850 con 0x7f8ccc145740 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43416/0 (socket says 172.21.15.90:43416) 2024-03-31T15:56:56.081 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 -- 172.21.15.90:0/4035461731 learned_addr learned my addr 172.21.15.90:0/4035461731 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8ccbe00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cc3e00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 -- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 msgr2=0x7f8ccc0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc0d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 -- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ccc15f9a0 msgr2=0x7f8ccc0c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 -- 172.21.15.90:0/4035461731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ccc1f8690 con 0x7f8ccc15fe20 2024-03-31T15:56:56.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd0a00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c7a0 tx=0x7f8cc400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4018070 con 0x7f8ccc15fe20 2024-03-31T15:56:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc400a640 con 0x7f8ccc15fe20 2024-03-31T15:56:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ccc1f8970 con 0x7f8ccc15fe20 2024-03-31T15:56:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400aab0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ccc1f8f30 con 0x7f8ccc15fe20 2024-03-31T15:56:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ccc068d00 con 0x7f8ccc15fe20 2024-03-31T15:56:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.087+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8cc401b070 con 0x7f8ccc15fe20 2024-03-31T15:56:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.091+0000 7f8cc8c00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 0x7f8ca806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.091+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f8cc4086190 con 0x7f8ccc15fe20 2024-03-31T15:56:56.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.091+0000 7f8ccbe00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 0x7f8ca806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.091+0000 7f8ccbe00640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 0x7f8ca806a840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009ac0 tx=0x7f8cbc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.095+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8cc4055ed0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ccc1f9100 con 0x7f8ccc15fe20 2024-03-31T15:56:56.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cc8c00640 1 -- 172.21.15.90:0/4035461731 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7f8cc405a1d0 con 0x7f8ccc15fe20 2024-03-31T15:56:56.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:56.249 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 msgr2=0x7f8ca806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 0x7f8ca806a840 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009ac0 tx=0x7f8cbc009190 comp rx=0 tx=0).stop 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 msgr2=0x7f8ccc0ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400c7a0 tx=0x7f8cc400cc60 comp rx=0 tx=0).stop 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 shutdown_connections 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8ca8068380 0x7f8ca806a840 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ccc15fe20 0x7f8ccc0ca450 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ccc15f9a0 0x7f8ccc0c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 --2- 172.21.15.90:0/4035461731 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8ccc145740 0x7f8ccc0d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 >> 172.21.15.90:0/4035461731 conn(0x7f8ccc0cd660 msgr2=0x7f8ccc144870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 shutdown_connections 2024-03-31T15:56:56.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.255+0000 7f8cd1400640 1 -- 172.21.15.90:0/4035461731 wait complete. 2024-03-31T15:56:56.271 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:56:56.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.411+0000 7fd6b2e00640 1 Processor -- start 2024-03-31T15:56:56.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.411+0000 7fd6b2e00640 1 -- start start 2024-03-31T15:56:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b2e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6ac15fd00 0x7fd6ac0d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b2e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b2e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac160350 con 0x7fd6ac15c120 2024-03-31T15:56:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b2e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac160bf0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.410 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b2e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac160630 con 0x7fd6ac0cca30 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6ab400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6ac15fd00 0x7fd6ac0d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:34472/0 (socket says 172.21.15.90:34472) 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6abe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd6ac15c120 0x7fd6ac0b26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:40510/0 (socket says 172.21.15.90:40510) 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 -- 172.21.15.90:0/4280505246 learned_addr learned my addr 172.21.15.90:0/4280505246 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 -- 172.21.15.90:0/4280505246 >> v2:172.21.15.102:3300/0 conn(0x7fd6ac15fd00 msgr2=0x7fd6ac0d2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 --2- 172.21.15.90:0/4280505246 >> v2:172.21.15.102:3300/0 conn(0x7fd6ac15fd00 0x7fd6ac0d2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 -- 172.21.15.90:0/4280505246 >> v1:172.21.15.102:6789/0 conn(0x7fd6ac15c120 legacy=0x7fd6ac0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.411 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.415+0000 7fd6b0e00640 1 -- 172.21.15.90:0/4280505246 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac160910 con 0x7fd6ac0cca30 2024-03-31T15:56:56.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b0e00640 1 --2- 172.21.15.90:0/4280505246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009e50 tx=0x7fd6a401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23059aa67dc28eda server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6aaa00640 1 -- 172.21.15.90:0/4280505246 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40a5770 con 0x7fd6ac0cca30 2024-03-31T15:56:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6aaa00640 1 -- 172.21.15.90:0/4280505246 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a40a5910 con 0x7fd6ac0cca30 2024-03-31T15:56:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6aaa00640 1 -- 172.21.15.90:0/4280505246 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40a5ce0 con 0x7fd6ac0cca30 2024-03-31T15:56:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 -- 172.21.15.90:0/4280505246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 msgr2=0x7fd6ac0ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/4280505246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009e50 tx=0x7fd6a401c9a0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 -- 172.21.15.90:0/4280505246 shutdown_connections 2024-03-31T15:56:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/4280505246 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac0ccea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/4280505246 >> v2:172.21.15.102:3300/0 conn(0x7fd6ac15fd00 0x7fd6ac0d2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 -- 172.21.15.90:0/4280505246 >> 172.21.15.90:0/4280505246 conn(0x7fd6ac09aa40 msgr2=0x7fd6ac09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 -- 172.21.15.90:0/4280505246 shutdown_connections 2024-03-31T15:56:56.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.419+0000 7fd6b2e00640 1 -- 172.21.15.90:0/4280505246 wait complete. 2024-03-31T15:56:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 Processor -- start 2024-03-31T15:56:56.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- start start 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac1591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 0x7fd6ac0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac1691c0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac169040 con 0x7fd6ac15c120 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac169340 con 0x7fd6ac0cca30 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43432/0 (socket says 172.21.15.90:43432) 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 -- 172.21.15.90:0/3160397601 learned_addr learned my addr 172.21.15.90:0/3160397601 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6ab400640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 0x7fd6ac0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6abe00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac1591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 -- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 msgr2=0x7fd6ac1591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac1591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 -- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 msgr2=0x7fd6ac0c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 0x7fd6ac0c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 -- 172.21.15.90:0/3160397601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac156030 con 0x7fd6ac15fd00 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6ab400640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 0x7fd6ac0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6abe00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:56.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b0e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd6a40af040 tx=0x7fd6a4019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40bc020 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a40b8670 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40a5770 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac1597b0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac159d70 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.423+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ac0686d0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.427+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd6a40bf020 con 0x7fd6ac15fd00 2024-03-31T15:56:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.427+0000 7fd6a3e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 0x7fd68c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.427+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fd6a4121ac0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.427+0000 7fd6abe00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 0x7fd68c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.427+0000 7fd6abe00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 0x7fd68c06a840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd69c007fd0 tx=0x7fd69c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.431+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd6a40c6030 con 0x7fd6ac15fd00 2024-03-31T15:56:56.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.619+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6ac070ae0 con 0x7fd6ac15fd00 2024-03-31T15:56:56.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.619+0000 7fd6a3e00640 1 -- 172.21.15.90:0/3160397601 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7fd6a4016620 con 0x7fd6ac15fd00 2024-03-31T15:56:56.616 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:56.616 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:56.616 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 msgr2=0x7fd68c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 0x7fd68c06a840 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd69c007fd0 tx=0x7fd69c009040 comp rx=0 tx=0).stop 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 msgr2=0x7fd6ac155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd6a40af040 tx=0x7fd6a4019740 comp rx=0 tx=0).stop 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 shutdown_connections 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd68c068380 0x7fd68c06a840 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd6ac15fd00 0x7fd6ac155af0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6ac15c120 0x7fd6ac0c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 --2- 172.21.15.90:0/3160397601 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd6ac0cca30 0x7fd6ac1591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 >> 172.21.15.90:0/3160397601 conn(0x7fd6ac09aa40 msgr2=0x7fd6ac156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 shutdown_connections 2024-03-31T15:56:56.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.623+0000 7fd6b2e00640 1 -- 172.21.15.90:0/3160397601 wait complete. 2024-03-31T15:56:56.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:56:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 Processor -- start 2024-03-31T15:56:56.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 -- start start 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5ec815e1b0 0x7f5ec8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5ec806c7b0 0x7f5ec806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec815f780 con 0x7f5ec806d5f0 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f5ec81622b0 con 0x7f5ec815e1b0 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ecdc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5ec806da20 con 0x7f5ec806c7b0 2024-03-31T15:56:56.775 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ec6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f5ec815e1b0 0x7f5ec8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ec7400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5ec806d5f0 0x7f5ec815e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57268/0 (socket says 172.21.15.90:57268) 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ec7400640 1 -- 172.21.15.90:0/278404195 learned_addr learned my addr 172.21.15.90:0/278404195 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.779+0000 7f5ec6000640 1 --2- 172.21.15.90:0/278404195 >> v2:172.21.15.102:3300/0 conn(0x7f5ec806c7b0 0x7f5ec806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.783+0000 7f5ec6a00640 1 -- 172.21.15.90:0/278404195 >> v2:172.21.15.102:3300/0 conn(0x7f5ec806c7b0 msgr2=0x7f5ec806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.783+0000 7f5ec6a00640 1 --2- 172.21.15.90:0/278404195 >> v2:172.21.15.102:3300/0 conn(0x7f5ec806c7b0 0x7f5ec806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.783+0000 7f5ec6a00640 1 -- 172.21.15.90:0/278404195 >> v1:172.21.15.90:6789/0 conn(0x7f5ec806d5f0 legacy=0x7f5ec815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.776 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.783+0000 7f5ec6a00640 1 -- 172.21.15.90:0/278404195 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec815f1c0 con 0x7f5ec815e1b0 2024-03-31T15:56:56.780 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ec6a00640 1 --2- 172.21.15.90:0/278404195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec815e1b0 0x7f5ec8162740 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5ec806ed50 tx=0x7f5eb801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d470be531a1fba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ec5600640 1 -- 172.21.15.90:0/278404195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80b1070 con 0x7f5ec815e1b0 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ec5600640 1 -- 172.21.15.90:0/278404195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eb80ac620 con 0x7f5ec815e1b0 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ec5600640 1 -- 172.21.15.90:0/278404195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80ac9f0 con 0x7f5ec815e1b0 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 -- 172.21.15.90:0/278404195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec815e1b0 msgr2=0x7f5ec8162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/278404195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec815e1b0 0x7f5ec8162740 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5ec806ed50 tx=0x7f5eb801c3c0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 -- 172.21.15.90:0/278404195 shutdown_connections 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/278404195 >> v2:172.21.15.102:3300/0 conn(0x7f5ec806c7b0 0x7f5ec806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/278404195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec815e1b0 0x7f5ec8162740 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 -- 172.21.15.90:0/278404195 >> 172.21.15.90:0/278404195 conn(0x7f5ec809aa40 msgr2=0x7f5ec809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 -- 172.21.15.90:0/278404195 shutdown_connections 2024-03-31T15:56:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 -- 172.21.15.90:0/278404195 wait complete. 2024-03-31T15:56:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.787+0000 7f5ecdc00640 1 Processor -- start 2024-03-31T15:56:56.783 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- start start 2024-03-31T15:56:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5ec806c7b0 0x7f5ec815a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec815e1b0 0x7f5ec80c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec80d2940 con 0x7f5ec806d5f0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec80d27c0 con 0x7f5ec815e1b0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5ec80d2ac0 con 0x7f5ec806c7b0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43452/0 (socket says 172.21.15.90:43452) 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 -- 172.21.15.90:0/4096047266 learned_addr learned my addr 172.21.15.90:0/4096047266 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6a00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5ec806c7b0 0x7f5ec815a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec7400640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec815e1b0 0x7f5ec80c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 -- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5ec806c7b0 msgr2=0x7f5ec815a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5ec806c7b0 0x7f5ec815a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 -- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec815e1b0 msgr2=0x7f5ec80c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec815e1b0 0x7f5ec80c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 -- 172.21.15.90:0/4096047266 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec815aa70 con 0x7f5ec806d5f0 2024-03-31T15:56:56.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ec6000640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400e8a0 tx=0x7f5eb400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb4017070 con 0x7f5ec806d5f0 2024-03-31T15:56:56.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eb400c940 con 0x7f5ec806d5f0 2024-03-31T15:56:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb400ccf0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec80703d0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec8070990 con 0x7f5ec806d5f0 2024-03-31T15:56:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.791+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec80686d0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.795+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5eb401a070 con 0x7f5ec806d5f0 2024-03-31T15:56:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.795+0000 7f5ebea00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 0x7f5ea806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.795+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f5eb4086db0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.795+0000 7f5ec6a00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 0x7f5ea806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:56.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.795+0000 7f5ec6a00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 0x7f5ea806a840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80ae000 tx=0x7f5eb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:56.792 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.799+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5eb4056af0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.987+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ec80b1b60 con 0x7f5ec806d5f0 2024-03-31T15:56:56.984 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ebea00640 1 -- 172.21.15.90:0/4096047266 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5184 (secure 0 0 0) 0x7f5eb405adf0 con 0x7f5ec806d5f0 2024-03-31T15:56:56.984 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:56.984 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-31T15:56:26.396713+0000","modified":"2024-03-31T15:56:29.426781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6213":{"gid":6213,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/1583024065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":1583024065},{"type":"v1","addr":"172.21.15.90:6835","nonce":1583024065}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6213,"qdb_cluster":[6213]},"id":3}]} 2024-03-31T15:56:56.984 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 27 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 msgr2=0x7f5ea806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 0x7f5ea806a840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80ae000 tx=0x7f5eb8008040 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 msgr2=0x7f5ec80c6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400e8a0 tx=0x7f5eb400eca0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 shutdown_connections 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ea8068380 0x7f5ea806a840 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec815e1b0 0x7f5ec80c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ec806d5f0 0x7f5ec80c6430 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 --2- 172.21.15.90:0/4096047266 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5ec806c7b0 0x7f5ec815a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:56.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 >> 172.21.15.90:0/4096047266 conn(0x7f5ec809aa40 msgr2=0x7f5ec815f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:56.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 shutdown_connections 2024-03-31T15:56:56.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:56.991+0000 7f5ecdc00640 1 -- 172.21.15.90:0/4096047266 wait complete. 2024-03-31T15:56:57.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:56:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.147+0000 7fd50ac00640 1 Processor -- start 2024-03-31T15:56:57.143 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.147+0000 7fd50ac00640 1 -- start start 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd50ac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504072c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd50ac00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50406e9f0 con 0x7fd5040cfbe0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd50ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5040b4340 con 0x7fd50406b5f0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd50ac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd504073290 con 0x7fd504161500 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd508e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5040cfbe0 0x7fd5040b3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57272/0 (socket says 172.21.15.90:57272) 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd508e00640 1 -- 172.21.15.90:0/1915623898 learned_addr learned my addr 172.21.15.90:0/1915623898 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd509800640 1 --2- 172.21.15.90:0/1915623898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2840524934 0 0) 0x7fd50406e9f0 con 0x7fd5040cfbe0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4ec003530 con 0x7fd5040cfbe0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782546034 0 0) 0x7fd5040b4340 con 0x7fd50406b5f0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd50406e9f0 con 0x7fd50406b5f0 2024-03-31T15:56:57.145 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd509800640 1 -- 172.21.15.90:0/1915623898 >> v1:172.21.15.102:6789/0 conn(0x7fd50406b5f0 legacy=0x7fd50406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd509800640 1 -- 172.21.15.90:0/1915623898 >> v1:172.21.15.90:6789/0 conn(0x7fd5040cfbe0 legacy=0x7fd5040b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.151+0000 7fd509800640 1 -- 172.21.15.90:0/1915623898 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50406ec90 con 0x7fd504161500 2024-03-31T15:56:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd509800640 1 --2- 172.21.15.90:0/1915623898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504072c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097f0 tx=0x7fd4f801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f00a49e826cedd6b server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80b0070 con 0x7fd504161500 2024-03-31T15:56:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f80ab600 con 0x7fd504161500 2024-03-31T15:56:57.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd503e00640 1 -- 172.21.15.90:0/1915623898 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80ab9d0 con 0x7fd504161500 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- 172.21.15.90:0/1915623898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 msgr2=0x7fd504072c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 --2- 172.21.15.90:0/1915623898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504072c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80097f0 tx=0x7fd4f801c450 comp rx=0 tx=0).stop 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- 172.21.15.90:0/1915623898 shutdown_connections 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 --2- 172.21.15.90:0/1915623898 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504072c00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- 172.21.15.90:0/1915623898 >> 172.21.15.90:0/1915623898 conn(0x7fd50406c700 msgr2=0x7fd50406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- 172.21.15.90:0/1915623898 shutdown_connections 2024-03-31T15:56:57.149 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- 172.21.15.90:0/1915623898 wait complete. 2024-03-31T15:56:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 Processor -- start 2024-03-31T15:56:57.150 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- start start 2024-03-31T15:56:57.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd50406b5f0 0x7fd50414f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd5040d0140 con 0x7fd5040cfbe0 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5040cffc0 con 0x7fd50406b5f0 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.155+0000 7fd50ac00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fd50416c540 con 0x7fd504161500 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43468/0 (socket says 172.21.15.90:43468) 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 -- 172.21.15.90:0/684436626 learned_addr learned my addr 172.21.15.90:0/684436626 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd509800640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd50406b5f0 0x7fd50414f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd50a200640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 -- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 msgr2=0x7fd504150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 -- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd50406b5f0 msgr2=0x7fd50414f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd50406b5f0 0x7fd50414f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 -- 172.21.15.90:0/684436626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5041603e0 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd508e00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c4d0 tx=0x7fd4f000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0017070 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f0005cf0 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00072b0 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5040b2a90 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5040b3050 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd504068d00 con 0x7fd5040cfbe0 2024-03-31T15:56:57.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.159+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd4f001a070 con 0x7fd5040cfbe0 2024-03-31T15:56:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.163+0000 7fd501600640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 0x7fd4e406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.163+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7fd4f0085f50 con 0x7fd5040cfbe0 2024-03-31T15:56:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.163+0000 7fd509800640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 0x7fd4e406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.163+0000 7fd509800640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 0x7fd4e406a840 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80ad000 tx=0x7fd4f80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.159 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.167+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fd4f0055c90 con 0x7fd5040cfbe0 2024-03-31T15:56:57.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.351+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5040b3220 con 0x7fd5040cfbe0 2024-03-31T15:56:57.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.351+0000 7fd501600640 1 -- 172.21.15.90:0/684436626 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16902 (secure 0 0 0) 0x7fd4f0059f90 con 0x7fd5040cfbe0 2024-03-31T15:56:57.347 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:56:57.347 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":55,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:56:28.034772+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-31T15:56:24.733192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-31T15:56:25.363340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 msgr2=0x7fd4e406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 0x7fd4e406a840 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd4f80ad000 tx=0x7fd4f80a6000 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 msgr2=0x7fd5040b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c4d0 tx=0x7fd4f000c990 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 shutdown_connections 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fd4e4068380 0x7fd4e406a840 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fd504161500 0x7fd504150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5040cfbe0 0x7fd5040b21e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 --2- 172.21.15.90:0/684436626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd50406b5f0 0x7fd50414f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.350 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 >> 172.21.15.90:0/684436626 conn(0x7fd50406c700 msgr2=0x7fd50415cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:57.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 shutdown_connections 2024-03-31T15:56:57.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.355+0000 7fd50ac00640 1 -- 172.21.15.90:0/684436626 wait complete. 2024-03-31T15:56:57.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-31T15:56:57.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 Processor -- start 2024-03-31T15:56:57.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 -- start start 2024-03-31T15:56:57.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f40cc15ec20 0x7f40cc0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 0x7f40cc0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc0b5250 con 0x7f40cc0addf0 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc158e90 con 0x7f40cc15ec20 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d3800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc0b5020 con 0x7f40cc0af890 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d1800640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f40cc0addf0 0x7f40cc15f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57274/0 (socket says 172.21.15.90:57274) 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d1800640 1 -- 172.21.15.90:0/3641979411 learned_addr learned my addr 172.21.15.90:0/3641979411 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d0e00640 1 --2- 172.21.15.90:0/3641979411 >> v2:172.21.15.102:3300/0 conn(0x7f40cc15ec20 0x7f40cc0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40cbe00640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 0x7f40cc0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40cb400640 1 -- 172.21.15.90:0/3641979411 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148516573 0 0) 0x7f40cc0b5250 con 0x7f40cc0addf0 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40cb400640 1 -- 172.21.15.90:0/3641979411 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b4003530 con 0x7f40cc0addf0 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d0e00640 1 -- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 msgr2=0x7f40cc0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d0e00640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 0x7f40cc0ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d0e00640 1 -- 172.21.15.90:0/3641979411 >> v1:172.21.15.90:6789/0 conn(0x7f40cc0addf0 legacy=0x7f40cc15f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40d0e00640 1 -- 172.21.15.90:0/3641979411 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0b5530 con 0x7f40cc15ec20 2024-03-31T15:56:57.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.579+0000 7f40cbe00640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 0x7f40cc0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:57.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d0e00640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc15ec20 0x7f40cc0b3710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40c00097c0 tx=0x7f40c001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d29beedaac5084e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40cb400640 1 -- 172.21.15.90:0/3641979411 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b2070 con 0x7f40cc15ec20 2024-03-31T15:56:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40cb400640 1 -- 172.21.15.90:0/3641979411 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00ad640 con 0x7f40cc15ec20 2024-03-31T15:56:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40cb400640 1 -- 172.21.15.90:0/3641979411 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00ada10 con 0x7f40cc15ec20 2024-03-31T15:56:57.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 -- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc15ec20 msgr2=0x7f40cc0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc15ec20 0x7f40cc0b3710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f40c00097c0 tx=0x7f40c001c4a0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 -- 172.21.15.90:0/3641979411 shutdown_connections 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc0af890 0x7f40cc0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 --2- 172.21.15.90:0/3641979411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc15ec20 0x7f40cc0b3710 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 -- 172.21.15.90:0/3641979411 >> 172.21.15.90:0/3641979411 conn(0x7f40cc0cd060 msgr2=0x7f40cc0cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 -- 172.21.15.90:0/3641979411 shutdown_connections 2024-03-31T15:56:57.579 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.583+0000 7f40d3800640 1 -- 172.21.15.90:0/3641979411 wait complete. 2024-03-31T15:56:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 Processor -- start 2024-03-31T15:56:57.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 -- start start 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc0af890 0x7f40cc06ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc15ec20 0x7f40cc13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc16af90 con 0x7f40cc0addf0 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc0afc70 con 0x7f40cc0af890 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d3800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc16b110 con 0x7f40cc15ec20 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43482/0 (socket says 172.21.15.90:43482) 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 -- 172.21.15.90:0/2023471852 learned_addr learned my addr 172.21.15.90:0/2023471852 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 -- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc15ec20 msgr2=0x7f40cc13fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc15ec20 0x7f40cc13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 -- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc0af890 msgr2=0x7f40cc06ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc0af890 0x7f40cc06ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.587+0000 7f40d0e00640 1 -- 172.21.15.90:0/2023471852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc06f180 con 0x7f40cc0addf0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40d0e00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f40c001a0d0 tx=0x7f40c0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00ba070 con 0x7f40cc0addf0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00b6410 con 0x7f40cc0addf0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b2040 con 0x7f40cc0addf0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc06f460 con 0x7f40cc0addf0 2024-03-31T15:56:57.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc157050 con 0x7f40cc0addf0 2024-03-31T15:56:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40cc068660 con 0x7f40cc0addf0 2024-03-31T15:56:57.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40c00bd070 con 0x7f40cc0addf0 2024-03-31T15:56:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.591+0000 7f40c8c00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 0x7f40ac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.595+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f40c0120940 con 0x7f40cc0addf0 2024-03-31T15:56:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.595+0000 7f40cbe00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 0x7f40ac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.595+0000 7f40cbe00640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 0x7f40ac06a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006e90 tx=0x7f40b8006de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.599+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f40c0016090 con 0x7f40cc0addf0 2024-03-31T15:56:57.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.787+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f40cc0af380 con 0x7f40cc0addf0 2024-03-31T15:56:57.782 INFO:tasks.ceph.mon.a.smithi090.stderr:2024-03-31T15:56:57.787+0000 7fc737200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-31T15:56:57.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40c8c00640 1 -- 172.21.15.90:0/2023471852 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) ==== 161+0+0 (secure 0 0 0) 0x7f40c00f0f20 con 0x7f40cc0addf0 2024-03-31T15:56:57.784 INFO:teuthology.orchestra.run.smithi090.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 msgr2=0x7f40ac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 0x7f40ac06a870 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006e90 tx=0x7f40b8006de0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 msgr2=0x7f40cc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f40c001a0d0 tx=0x7f40c0019240 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 shutdown_connections 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f40ac0683b0 0x7f40ac06a870 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f40cc15ec20 0x7f40cc13fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40cc0af890 0x7f40cc06ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 --2- 172.21.15.90:0/2023471852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f40cc0addf0 0x7f40cc06d9e0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.791+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 >> 172.21.15.90:0/2023471852 conn(0x7f40cc0cd060 msgr2=0x7f40cc152690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.795+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 shutdown_connections 2024-03-31T15:56:57.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.795+0000 7f40d3800640 1 -- 172.21.15.90:0/2023471852 wait complete. 2024-03-31T15:56:57.798 INFO:tasks.ceph.mds.a.smithi090.stderr: -2960> 2024-03-31T15:56:28.992+0000 7f1df4800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-31T15:56:57.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-31T15:56:57.851 INFO:tasks.ceph.mds.a.smithi090.stdout:starting mds.a at 2024-03-31T15:56:57.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.959+0000 7f772fe00640 1 Processor -- start 2024-03-31T15:56:57.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.959+0000 7f772fe00640 1 -- start start 2024-03-31T15:56:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772fe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7730161e10 0x7f77301621f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 0x7f773006d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772fe00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7730162730 con 0x7f77300d0a30 2024-03-31T15:56:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772fe00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f77300d0f90 con 0x7f7730161e10 2024-03-31T15:56:57.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f77300d1270 con 0x7f773015e4d0 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7730161e10 0x7f77301621f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7730161e10 0x7f77301621f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43506/0 (socket says 172.21.15.90:43506) 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 -- 172.21.15.90:0/1863455474 learned_addr learned my addr 172.21.15.90:0/1863455474 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772f400640 1 --1- 172.21.15.90:0/1863455474 >> v1:172.21.15.90:6789/0 conn(0x7f77300d0a30 0x7f77300ce800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57296/0 (socket says 172.21.15.90:57296) 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f7726000640 1 --2- 172.21.15.90:0/1863455474 >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 0x7f773006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 -- 172.21.15.90:0/1863455474 >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 msgr2=0x7f773006d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 --2- 172.21.15.90:0/1863455474 >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 0x7f773006d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 -- 172.21.15.90:0/1863455474 >> v1:172.21.15.90:6789/0 conn(0x7f77300d0a30 legacy=0x7f77300ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 -- 172.21.15.90:0/1863455474 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77300d1550 con 0x7f7730161e10 2024-03-31T15:56:57.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f7726000640 1 --2- 172.21.15.90:0/1863455474 >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 0x7f773006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:56:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772ea00640 1 --2- 172.21.15.90:0/1863455474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7730161e10 0x7f77301621f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f77300b48b0 tx=0x7f772001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c6ba7a3ae1b3b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.963+0000 7f772e000640 1 -- 172.21.15.90:0/1863455474 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200b1070 con 0x7f7730161e10 2024-03-31T15:56:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772e000640 1 -- 172.21.15.90:0/1863455474 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77200ac600 con 0x7f7730161e10 2024-03-31T15:56:57.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772e000640 1 -- 172.21.15.90:0/1863455474 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200ac9d0 con 0x7f7730161e10 2024-03-31T15:56:57.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- 172.21.15.90:0/1863455474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7730161e10 msgr2=0x7f77301621f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- 172.21.15.90:0/1863455474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7730161e10 0x7f77301621f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f77300b48b0 tx=0x7f772001c490 comp rx=0 tx=0).stop 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- 172.21.15.90:0/1863455474 shutdown_connections 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- 172.21.15.90:0/1863455474 >> v2:172.21.15.102:3300/0 conn(0x7f773015e4d0 0x7f773006d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- 172.21.15.90:0/1863455474 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7730161e10 0x7f77301621f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- 172.21.15.90:0/1863455474 >> 172.21.15.90:0/1863455474 conn(0x7f77300d1f80 msgr2=0x7f77300d2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- 172.21.15.90:0/1863455474 shutdown_connections 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- 172.21.15.90:0/1863455474 wait complete. 2024-03-31T15:56:57.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 Processor -- start 2024-03-31T15:56:57.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- start start 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 0x7f773015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 0x7f7730160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f773016b4e0 con 0x7f773015e4d0 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f773016b360 con 0x7f77300d0a30 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772fe00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f773016b660 con 0x7f7730161e10 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43512/0 (socket says 172.21.15.90:43512) 2024-03-31T15:56:57.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 -- 172.21.15.90:0/535569233 learned_addr learned my addr 172.21.15.90:0/535569233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772ea00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 0x7f773015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772f400640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 0x7f7730160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 -- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 msgr2=0x7f7730160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 0x7f7730160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 -- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 msgr2=0x7f773015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 0x7f773015c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f7726000640 1 -- 172.21.15.90:0/535569233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7730160a40 con 0x7f773015e4d0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772ea00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 0x7f773015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.967+0000 7f772f400640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 0x7f7730160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7726000640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f771800c7f0 tx=0x7f771800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718007a30 con 0x7f773015e4d0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7718007bd0 con 0x7f773015e4d0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77300c6260 con 0x7f773015e4d0 2024-03-31T15:56:57.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77180113f0 con 0x7f773015e4d0 2024-03-31T15:56:57.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7730156750 con 0x7f773015e4d0 2024-03-31T15:56:57.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77300685f0 con 0x7f773015e4d0 2024-03-31T15:56:57.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7718011590 con 0x7f773015e4d0 2024-03-31T15:56:57.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 0x7f770c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:57.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6187+0+0 (secure 0 0 0) 0x7f7718086df0 con 0x7f773015e4d0 2024-03-31T15:56:57.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.971+0000 7f772ea00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 0x7f770c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:57.969 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.975+0000 7f772ea00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 0x7f770c06a840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f77200ae000 tx=0x7f77200a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:57.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:57.975+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f7718015020 con 0x7f773015e4d0 2024-03-31T15:56:58.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.171+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f77300b4920 con 0x7f773015e4d0 2024-03-31T15:56:58.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.791+0000 7f7727400640 1 -- 172.21.15.90:0/535569233 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f77180571b0 con 0x7f773015e4d0 2024-03-31T15:56:58.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 msgr2=0x7f770c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 0x7f770c06a840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f77200ae000 tx=0x7f77200a7000 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 msgr2=0x7f773015fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f771800c7f0 tx=0x7f771800ccb0 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 shutdown_connections 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f770c068380 0x7f770c06a840 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7730161e10 0x7f7730160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773015e4d0 0x7f773015fed0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 --2- 172.21.15.90:0/535569233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77300d0a30 0x7f773015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 >> 172.21.15.90:0/535569233 conn(0x7f77300d1f80 msgr2=0x7f77300cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 shutdown_connections 2024-03-31T15:56:58.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:58.795+0000 7f772fe00640 1 -- 172.21.15.90:0/535569233 wait complete. 2024-03-31T15:56:58.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-31T15:56:59.000 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 Processor -- start 2024-03-31T15:56:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 -- start start 2024-03-31T15:56:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 0x7f122415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.001 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122415cd60 con 0x7f122415be10 2024-03-31T15:56:59.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f122415fdc0 con 0x7f1224160790 2024-03-31T15:56:59.002 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f122bc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f122406ee90 con 0x7f122406ea10 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f1229c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 0x7f122415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:34544/0 (socket says 172.21.15.90:34544) 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 -- 172.21.15.90:0/3973381005 learned_addr learned my addr 172.21.15.90:0/3973381005 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f1223e00640 1 -- 172.21.15.90:0/3973381005 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250088544 0 0) 0x7f122415cd60 con 0x7f122415be10 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f1223e00640 1 -- 172.21.15.90:0/3973381005 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1208003530 con 0x7f122415be10 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 -- 172.21.15.90:0/3973381005 >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 msgr2=0x7f122415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 --2- 172.21.15.90:0/3973381005 >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 0x7f122415c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 -- 172.21.15.90:0/3973381005 >> v1:172.21.15.102:6789/0 conn(0x7f122415be10 legacy=0x7f12240cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f121be00640 1 -- 172.21.15.90:0/3973381005 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122415d040 con 0x7f122406ea10 2024-03-31T15:56:59.003 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.007+0000 7f1229c00640 1 --2- 172.21.15.90:0/3973381005 >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 0x7f122415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:56:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f121be00640 1 --2- 172.21.15.90:0/3973381005 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f120c009870 tx=0x7f120c01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a06221ff13c6026a server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:59.005 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f1223e00640 1 -- 172.21.15.90:0/3973381005 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c0b1070 con 0x7f122406ea10 2024-03-31T15:56:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f1223e00640 1 -- 172.21.15.90:0/3973381005 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f120c0a8050 con 0x7f122406ea10 2024-03-31T15:56:59.006 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f1223e00640 1 -- 172.21.15.90:0/3973381005 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c0ac860 con 0x7f122406ea10 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 -- 172.21.15.90:0/3973381005 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 msgr2=0x7f1224160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 --2- 172.21.15.90:0/3973381005 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f120c009870 tx=0x7f120c01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 -- 172.21.15.90:0/3973381005 shutdown_connections 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 --2- 172.21.15.90:0/3973381005 >> v2:172.21.15.102:3300/0 conn(0x7f1224160790 0x7f122415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 --2- 172.21.15.90:0/3973381005 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f1224160250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 -- 172.21.15.90:0/3973381005 >> 172.21.15.90:0/3973381005 conn(0x7f12240b1880 msgr2=0x7f12240b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 -- 172.21.15.90:0/3973381005 shutdown_connections 2024-03-31T15:56:59.007 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.011+0000 7f122bc00640 1 -- 172.21.15.90:0/3973381005 wait complete. 2024-03-31T15:56:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 Processor -- start 2024-03-31T15:56:59.008 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- start start 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f12241597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122415be10 0x7f12240cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f12241692e0 con 0x7f1224160790 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1224169160 con 0x7f122415be10 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1224169460 con 0x7f122406ea10 2024-03-31T15:56:59.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43514/0 (socket says 172.21.15.90:43514) 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 -- 172.21.15.90:0/3048108369 learned_addr learned my addr 172.21.15.90:0/3048108369 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229200640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f12241597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f121be00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122415be10 0x7f12240cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 -- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 msgr2=0x7f12241597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f12241597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 -- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122415be10 msgr2=0x7f12240cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122415be10 0x7f12240cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 -- 172.21.15.90:0/3048108369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12240cf8a0 con 0x7f1224160790 2024-03-31T15:56:59.011 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1229c00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e60 tx=0x7f121c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c018030 con 0x7f1224160790 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121c00cdf0 con 0x7f1224160790 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12241565c0 con 0x7f1224160790 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121c003bc0 con 0x7f1224160790 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.015+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12241fb1b0 con 0x7f1224160790 2024-03-31T15:56:59.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12240686d0 con 0x7f1224160790 2024-03-31T15:56:59.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f121c01b070 con 0x7f1224160790 2024-03-31T15:56:59.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f1221600640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 0x7f120006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:56:59.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6187+0+0 (secure 0 0 0) 0x7f121c087440 con 0x7f1224160790 2024-03-31T15:56:59.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f1229200640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 0x7f120006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:56:59.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.019+0000 7f1229200640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 0x7f120006a840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f12140017c0 tx=0x7f1214009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:56:59.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.023+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f121c022070 con 0x7f1224160790 2024-03-31T15:56:59.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:56:59.187+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f12240b4990 con 0x7f1224160790 2024-03-31T15:57:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.159+0000 7f1221600640 1 -- 172.21.15.90:0/3048108369 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) ==== 181+0+0 (secure 0 0 0) 0x7f121c010040 con 0x7f1224160790 2024-03-31T15:57:00.153 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_metadata' removed 2024-03-31T15:57:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 msgr2=0x7f120006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 0x7f120006a840 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f12140017c0 tx=0x7f1214009040 comp rx=0 tx=0).stop 2024-03-31T15:57:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 msgr2=0x7f12240ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:00.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f121c009e60 tx=0x7f121c00c490 comp rx=0 tx=0).stop 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 shutdown_connections 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1200068380 0x7f120006a840 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1224160790 0x7f12240ca360 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122415be10 0x7f12240cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 --2- 172.21.15.90:0/3048108369 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f122406ea10 0x7f12241597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 >> 172.21.15.90:0/3048108369 conn(0x7f12240b1880 msgr2=0x7f12241575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 shutdown_connections 2024-03-31T15:57:00.157 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.163+0000 7f122bc00640 1 -- 172.21.15.90:0/3048108369 wait complete. 2024-03-31T15:57:00.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-31T15:57:00.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 Processor -- start 2024-03-31T15:57:00.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 -- start start 2024-03-31T15:57:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f815be10 0x7f61f806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61f80cb630 0x7f61f80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f61f8160250 0x7f61f81606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f61f815fdc0 con 0x7f61f80cb630 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f61f815cd60 con 0x7f61f8160250 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61fe400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f61f815d040 con 0x7f61f815be10 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61f80cb630 0x7f61f80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f61f80cb630 0x7f61f80cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43528/0 (socket says 172.21.15.90:43528) 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 -- 172.21.15.90:0/142486991 learned_addr learned my addr 172.21.15.90:0/142486991 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 -- 172.21.15.90:0/142486991 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f815be10 msgr2=0x7f61f806ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 --2- 172.21.15.90:0/142486991 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f815be10 0x7f61f806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 -- 172.21.15.90:0/142486991 >> v2:172.21.15.102:3300/0 conn(0x7f61f8160250 msgr2=0x7f61f81606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 --2- 172.21.15.90:0/142486991 >> v2:172.21.15.102:3300/0 conn(0x7f61f8160250 0x7f61f81606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.339+0000 7f61eea00640 1 -- 172.21.15.90:0/142486991 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f815d320 con 0x7f61f80cb630 2024-03-31T15:57:00.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61eea00640 1 --2- 172.21.15.90:0/142486991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f80cbab0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f61f80af7a0 tx=0x7f61e001c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d99abcdd156c59f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61f6a00640 1 -- 172.21.15.90:0/142486991 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e00b0070 con 0x7f61f80cb630 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61f6a00640 1 -- 172.21.15.90:0/142486991 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e00ab600 con 0x7f61f80cb630 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61f6a00640 1 -- 172.21.15.90:0/142486991 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e00ab9d0 con 0x7f61f80cb630 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 -- 172.21.15.90:0/142486991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 msgr2=0x7f61f80cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 --2- 172.21.15.90:0/142486991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f80cbab0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f61f80af7a0 tx=0x7f61e001c1d0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 -- 172.21.15.90:0/142486991 shutdown_connections 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 --2- 172.21.15.90:0/142486991 >> v2:172.21.15.102:3300/0 conn(0x7f61f8160250 0x7f61f81606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 --2- 172.21.15.90:0/142486991 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f80cbab0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 --2- 172.21.15.90:0/142486991 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f815be10 0x7f61f806ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 -- 172.21.15.90:0/142486991 >> 172.21.15.90:0/142486991 conn(0x7f61f80b1880 msgr2=0x7f61f80b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 -- 172.21.15.90:0/142486991 shutdown_connections 2024-03-31T15:57:00.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.343+0000 7f61fe400640 1 -- 172.21.15.90:0/142486991 wait complete. 2024-03-31T15:57:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 Processor -- start 2024-03-31T15:57:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- start start 2024-03-31T15:57:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f815be10 0x7f61f814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f8160250 0x7f61f8156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61f80ca500 con 0x7f61f80cb630 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f61f80ca380 con 0x7f61f815be10 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f61f80ca680 con 0x7f61f8160250 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43534/0 (socket says 172.21.15.90:43534) 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 -- 172.21.15.90:0/4220082800 learned_addr learned my addr 172.21.15.90:0/4220082800 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 -- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f8160250 msgr2=0x7f61f8156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f8160250 0x7f61f8156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.345 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 -- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f815be10 msgr2=0x7f61f814f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f815be10 0x7f61f814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 -- 172.21.15.90:0/4220082800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f81599c0 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61f7400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f61e800ba80 tx=0x7f61e800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8019070 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e800cc20 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f814ff20 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8014440 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f8159650 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.347+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f80686d0 con 0x7f61f80cb630 2024-03-31T15:57:00.346 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.351+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61e801c070 con 0x7f61f80cb630 2024-03-31T15:57:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.351+0000 7f61efe00640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 0x7f61d406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.351+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5749+0+0 (secure 0 0 0) 0x7f61e8086650 con 0x7f61f80cb630 2024-03-31T15:57:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.351+0000 7f61eea00640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 0x7f61d406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:00.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.351+0000 7f61eea00640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 0x7f61d406a840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f61e00ad000 tx=0x7f61e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:00.349 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.355+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f61e8056590 con 0x7f61f80cb630 2024-03-31T15:57:00.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:00.511+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f61f806e6a0 con 0x7f61f80cb630 2024-03-31T15:57:01.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.159+0000 7f61efe00640 1 -- 172.21.15.90:0/4220082800 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) ==== 169+0+0 (secure 0 0 0) 0x7f61e805a890 con 0x7f61f80cb630 2024-03-31T15:57:01.155 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'cephfs_data' removed 2024-03-31T15:57:01.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.163+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 msgr2=0x7f61d406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.163+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 0x7f61d406a840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f61e00ad000 tx=0x7f61e0008040 comp rx=0 tx=0).stop 2024-03-31T15:57:01.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 msgr2=0x7f61f8150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f61e800ba80 tx=0x7f61e800bf40 comp rx=0 tx=0).stop 2024-03-31T15:57:01.161 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 shutdown_connections 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f61d4068380 0x7f61d406a840 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f61f8160250 0x7f61f8156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f815be10 0x7f61f814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 --2- 172.21.15.90:0/4220082800 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f80cb630 0x7f61f8150200 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 >> 172.21.15.90:0/4220082800 conn(0x7f61f80b1880 msgr2=0x7f61f80af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 shutdown_connections 2024-03-31T15:57:01.162 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.167+0000 7f61fe400640 1 -- 172.21.15.90:0/4220082800 wait complete. 2024-03-31T15:57:01.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-31T15:57:01.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 Processor -- start 2024-03-31T15:57:01.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 -- start start 2024-03-31T15:57:01.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fed1015b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed10072a60 con 0x7fed10071940 2024-03-31T15:57:01.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed100aeac0 con 0x7fed1015b040 2024-03-31T15:57:01.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed16600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fed100ae890 con 0x7fed100725a0 2024-03-31T15:57:01.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed0fe00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fed10071940 0x7fed100b3a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57312/0 (socket says 172.21.15.90:57312) 2024-03-31T15:57:01.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed0fe00640 1 -- 172.21.15.90:0/2652575845 learned_addr learned my addr 172.21.15.90:0/2652575845 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed06a00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fed1015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2142956818 0 0) 0x7fed10072a60 con 0x7fed10071940 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf4003530 con 0x7fed10071940 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.331+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4736590 0 0) 0x7fed100aeac0 con 0x7fed1015b040 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed10072a60 con 0x7fed1015b040 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2905832021 0 0) 0x7fecf4003530 con 0x7fed10071940 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 msgr2=0x7fed1015b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fed1015b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 >> v1:172.21.15.102:6789/0 conn(0x7fed1015b040 legacy=0x7fed10071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed100aeda0 con 0x7fed10071940 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fed08001a00 con 0x7fed10071940 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 >> v1:172.21.15.90:6789/0 conn(0x7fed10071940 legacy=0x7fed100b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecf4003b80 0x7fecf4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fecf4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecf4004d50 0x7fecf40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fecf400dc90 con 0x7fecf4003b80 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fecf400de40 con 0x7fecf4004d50 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fecf400e040 con 0x7fed100725a0 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecf4003b80 0x7fecf4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed06a00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fecf4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0fe00640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecf4004d50 0x7fecf40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 -- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 msgr2=0x7fecf4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:01.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fed100725a0 0x7fecf4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 -- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecf4004d50 msgr2=0x7fecf40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecf4004d50 0x7fecf40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 -- 172.21.15.90:0/2652575845 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf4003530 con 0x7fecf4003b80 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0f400640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecf4003b80 0x7fecf4003f80 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fed000018d0 tx=0x7fed0001c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3addea5c934c804 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.335+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00001520 con 0x7fecf4003b80 2024-03-31T15:57:01.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed000050d0 con 0x7fecf4003b80 2024-03-31T15:57:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0ea00640 1 -- 172.21.15.90:0/2652575845 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed000054a0 con 0x7fecf4003b80 2024-03-31T15:57:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- 172.21.15.90:0/2652575845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecf4003b80 msgr2=0x7fecf4003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 --2- 172.21.15.90:0/2652575845 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fecf4003b80 0x7fecf4003f80 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fed000018d0 tx=0x7fed0001c580 comp rx=0 tx=0).stop 2024-03-31T15:57:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0fe00640 1 -- 172.21.15.90:0/2652575845 reap_dead start 2024-03-31T15:57:01.332 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- 172.21.15.90:0/2652575845 shutdown_connections 2024-03-31T15:57:01.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- 172.21.15.90:0/2652575845 >> 172.21.15.90:0/2652575845 conn(0x7fed100b18b0 msgr2=0x7fed100b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:01.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- 172.21.15.90:0/2652575845 shutdown_connections 2024-03-31T15:57:01.333 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- 172.21.15.90:0/2652575845 wait complete. 2024-03-31T15:57:01.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 Processor -- start 2024-03-31T15:57:01.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- start start 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 0x7fed10146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101f3270 0x7fed10135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fed1006d870 con 0x7fed101469d0 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed100b35c0 con 0x7fed101f3270 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed16600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fed1006d9f0 con 0x7fecf4003b80 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 0x7fed10146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0f400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 0x7fed10146060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:34602/0 (socket says 172.21.15.90:34602) 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0fe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43552/0 (socket says 172.21.15.90:43552) 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed0f400640 1 -- 172.21.15.90:0/2116487361 learned_addr learned my addr 172.21.15.90:0/2116487361 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.339+0000 7fed06a00640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101f3270 0x7fed10135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 -- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 msgr2=0x7fed10146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 0x7fed10146060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 -- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101f3270 msgr2=0x7fed10135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101f3270 0x7fed10135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 -- 172.21.15.90:0/2116487361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed10136a90 con 0x7fed101469d0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0fe00640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fed10074000 tx=0x7fed08007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0800aad0 con 0x7fed101469d0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed08018070 con 0x7fed101469d0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed10136d70 con 0x7fed101469d0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed080133f0 con 0x7fed101469d0 2024-03-31T15:57:01.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed10137330 con 0x7fed101469d0 2024-03-31T15:57:01.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed100685f0 con 0x7fed101469d0 2024-03-31T15:57:01.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fed08013590 con 0x7fed101469d0 2024-03-31T15:57:01.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 0x7fecec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fed08086ce0 con 0x7fed101469d0 2024-03-31T15:57:01.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.343+0000 7fed0f400640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 0x7fecec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.347+0000 7fed0f400640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 0x7fecec06a840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed00005040 tx=0x7fed000a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.347+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fed08057370 con 0x7fed101469d0 2024-03-31T15:57:01.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.503+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fed100b2440 con 0x7fed101469d0 2024-03-31T15:57:01.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed07e00640 1 -- 172.21.15.90:0/2116487361 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7fed08056c70 con 0x7fed101469d0 2024-03-31T15:57:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 msgr2=0x7fecec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 0x7fecec06a840 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed00005040 tx=0x7fed000a6040 comp rx=0 tx=0).stop 2024-03-31T15:57:01.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 msgr2=0x7fed1014dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fed10074000 tx=0x7fed08007b10 comp rx=0 tx=0).stop 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 shutdown_connections 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fecec068380 0x7fecec06a840 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fed101469d0 0x7fed1014dbb0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed101f3270 0x7fed10135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 --2- 172.21.15.90:0/2116487361 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fecf4003b80 0x7fed10146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 >> 172.21.15.90:0/2116487361 conn(0x7fed100b18b0 msgr2=0x7fed10159bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 shutdown_connections 2024-03-31T15:57:01.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.507+0000 7fed16600640 1 -- 172.21.15.90:0/2116487361 wait complete. 2024-03-31T15:57:01.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-31T15:57:01.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.667+0000 7ff7eb200640 1 Processor -- start 2024-03-31T15:57:01.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.667+0000 7ff7eb200640 1 -- start start 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7eb200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff7ec15c070 0x7ff7ec15d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7eb200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec15d570 con 0x7ff7ec0ce780 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7eb200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec0d0c40 con 0x7ff7ec0d0500 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7eb200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ec15c450 con 0x7ff7ec15c070 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7ff7ec15c070 0x7ff7ec15d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.664 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff7ec0ce780 0x7ff7ec0cfdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:57326/0 (socket says 172.21.15.90:57326) 2024-03-31T15:57:01.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9400640 1 -- 172.21.15.90:0/2279221880 learned_addr learned my addr 172.21.15.90:0/2279221880 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:01.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9e00640 1 -- 172.21.15.90:0/2279221880 >> v1:172.21.15.102:6789/0 conn(0x7ff7ec0d0500 legacy=0x7ff7ec0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:01.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9e00640 1 -- 172.21.15.90:0/2279221880 >> v1:172.21.15.90:6789/0 conn(0x7ff7ec0ce780 legacy=0x7ff7ec0cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e8a00640 1 -- 172.21.15.90:0/2279221880 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101929631 0 0) 0x7ff7ec15d570 con 0x7ff7ec0ce780 2024-03-31T15:57:01.665 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.671+0000 7ff7e9e00640 1 -- 172.21.15.90:0/2279221880 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec0cec80 con 0x7ff7ec15c070 2024-03-31T15:57:01.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7e9e00640 1 --2- 172.21.15.90:0/2279221880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec15c070 0x7ff7ec15d030 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0097c0 tx=0x7ff7dc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b21140324cfac1f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7e8a00640 1 -- 172.21.15.90:0/2279221880 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b5070 con 0x7ff7ec15c070 2024-03-31T15:57:01.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7e8a00640 1 -- 172.21.15.90:0/2279221880 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0a8950 con 0x7ff7ec15c070 2024-03-31T15:57:01.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7e8a00640 1 -- 172.21.15.90:0/2279221880 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0a8d20 con 0x7ff7ec15c070 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- 172.21.15.90:0/2279221880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec15c070 msgr2=0x7ff7ec15d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 --2- 172.21.15.90:0/2279221880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec15c070 0x7ff7ec15d030 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0097c0 tx=0x7ff7dc01ec50 comp rx=0 tx=0).stop 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- 172.21.15.90:0/2279221880 shutdown_connections 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 --2- 172.21.15.90:0/2279221880 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec15c070 0x7ff7ec15d030 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- 172.21.15.90:0/2279221880 >> 172.21.15.90:0/2279221880 conn(0x7ff7ec0d1d70 msgr2=0x7ff7ec0d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- 172.21.15.90:0/2279221880 shutdown_connections 2024-03-31T15:57:01.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- 172.21.15.90:0/2279221880 wait complete. 2024-03-31T15:57:01.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 Processor -- start 2024-03-31T15:57:01.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 -- start start 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 0x7ff7ec1f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.675+0000 7ff7eb200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7ec15c070 0x7ff7ec0b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec0cf520 con 0x7ff7ec0ce780 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ec0cf240 con 0x7ff7ec0d0500 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ec0cc880 con 0x7ff7ec15c070 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:43566/0 (socket says 172.21.15.90:43566) 2024-03-31T15:57:01.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 -- 172.21.15.90:0/934834299 learned_addr learned my addr 172.21.15.90:0/934834299 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 -- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7ec15c070 msgr2=0x7ff7ec0b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7ec15c070 0x7ff7ec0b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 -- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 msgr2=0x7ff7ec1f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9400640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 0x7ff7ec1f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 0x7ff7ec1f1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 -- 172.21.15.90:0/934834299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec0c57f0 con 0x7ff7ec0ce780 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9400640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 0x7ff7ec1f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e9e00640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b3040 tx=0x7ff7dc01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b5040 con 0x7ff7ec0ce780 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0c0070 con 0x7ff7ec0ce780 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc0b0a80 con 0x7ff7ec0ce780 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec14e990 con 0x7ff7ec0ce780 2024-03-31T15:57:01.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ec0c6050 con 0x7ff7ec0ce780 2024-03-31T15:57:01.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ec0685f0 con 0x7ff7ec0ce780 2024-03-31T15:57:01.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.679+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff7dc0b0c20 con 0x7ff7ec0ce780 2024-03-31T15:57:01.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.683+0000 7ff7e2000640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 0x7ff7cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:01.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.683+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7ff7dc123ef0 con 0x7ff7ec0ce780 2024-03-31T15:57:01.676 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.683+0000 7ff7e9400640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 0x7ff7cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:01.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.683+0000 7ff7e9400640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 0x7ff7cc06a840 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8007fd0 tx=0x7ff7d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:01.679 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.687+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7ff7dc0187b0 con 0x7ff7ec0ce780 2024-03-31T15:57:01.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:01.907+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7ff7ec06c2e0 con 0x7ff7ec0ce780 2024-03-31T15:57:02.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7e2000640 1 -- 172.21.15.90:0/934834299 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v199) ==== 134+0+0 (secure 0 0 0) 0x7ff7dc0af3a0 con 0x7ff7ec0ce780 2024-03-31T15:57:02.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 msgr2=0x7ff7cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:02.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 0x7ff7cc06a840 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8007fd0 tx=0x7ff7d8009040 comp rx=0 tx=0).stop 2024-03-31T15:57:02.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 msgr2=0x7ff7ec1f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0b3040 tx=0x7ff7dc01b9f0 comp rx=0 tx=0).stop 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 shutdown_connections 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7ff7cc068380 0x7ff7cc06a840 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7ff7ec15c070 0x7ff7ec0b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7ec0d0500 0x7ff7ec1f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 --2- 172.21.15.90:0/934834299 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff7ec0ce780 0x7ff7ec1f1760 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 >> 172.21.15.90:0/934834299 conn(0x7ff7ec0d1d70 msgr2=0x7ff7ec15e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 shutdown_connections 2024-03-31T15:57:02.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:02.175+0000 7ff7eb200640 1 -- 172.21.15.90:0/934834299 wait complete. 2024-03-31T15:57:02.193 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-31T15:57:02.193 INFO:tasks.cephfs_test_runner: 2024-03-31T15:57:02.193 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner: 2024-03-31T15:57:02.194 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:Ran 2 tests in 141.003s 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-31T15:57:02.195 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-31T15:57:02.196 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-31T15:57:02.196 INFO:tasks.cephfs_test_runner: 2024-03-31T15:57:02.196 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:57:02.450 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b4145c849c554861b3f4c377652b2e95 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_d2271579d5e735cfd6d436b8349c004f06527cb0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-31T15:57:02.451 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-31T15:57:02.462 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-31T15:57:02.471 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-31T15:57:02.472 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:57:02.472 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:57:02.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:57:02.487 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-31T15:57:02.488 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:57:02.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354552k,nr_inodes=4088638,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3894 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-31T15:57:02.536 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:57:02.536 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:57:02.545 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:57:02.545 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-31T15:57:02.545 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-31T15:57:02.596 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354612k,nr_inodes=4088653,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9376 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275372k,nr_inodes=818843,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275376k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-31T15:57:02.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.597 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-31T15:57:02.649 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-31T15:57:02.649 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-31T15:57:02.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-31T15:57:02.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.714 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-31T15:57:02.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-31T15:57:02.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.960 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-03-31T15:57:02.969 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-31T15:57:02.969 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-31T15:57:02.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:02.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-31T15:57:03.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:03.033 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-31T15:57:03.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-31T15:57:03.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-03-31T15:57:03.278 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-31T15:57:03.288 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-31T15:57:03.288 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:57:03.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.431+0000 7f0d30e00640 1 Processor -- start 2024-03-31T15:57:03.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.431+0000 7f0d30e00640 1 -- start start 2024-03-31T15:57:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d30e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d2c0ccdb0 0x7f0d2c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d30e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c06b820 0x7f0d2c06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d30e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d2c0b34f0 con 0x7f0d2c0cc9d0 2024-03-31T15:57:03.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d30e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0d2c0b3ab0 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d30e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2c141ba0 con 0x7f0d2c06b820 2024-03-31T15:57:03.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d2a800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c06b820 0x7f0d2c06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0d2c0cc9d0 0x7f0d2c142030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42452/0 (socket says 172.21.15.90:42452) 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29e00640 1 -- 172.21.15.90:0/4128144584 learned_addr learned my addr 172.21.15.90:0/4128144584 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d2c0ccdb0 0x7f0d2c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29400640 1 -- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c06b820 msgr2=0x7f0d2c06bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29400640 1 --2- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c06b820 0x7f0d2c06bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29400640 1 -- 172.21.15.90:0/4128144584 >> v1:172.21.15.102:6789/0 conn(0x7f0d2c0cc9d0 legacy=0x7f0d2c142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.435+0000 7f0d29400640 1 -- 172.21.15.90:0/4128144584 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c0b37d0 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d29400640 1 --2- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c0b2b80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097c0 tx=0x7f0d1801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d350f94985688f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d28a00640 1 -- 172.21.15.90:0/4128144584 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d180b0070 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d28a00640 1 -- 172.21.15.90:0/4128144584 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d180ab600 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d28a00640 1 -- 172.21.15.90:0/4128144584 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d180ab9d0 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.433 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c0ccdb0 msgr2=0x7f0d2c0b2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 --2- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c0b2b80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097c0 tx=0x7f0d1801c410 comp rx=0 tx=0).stop 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- 172.21.15.90:0/4128144584 shutdown_connections 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 --2- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c06b820 0x7f0d2c06bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 --2- 172.21.15.90:0/4128144584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c0b2b80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- 172.21.15.90:0/4128144584 >> 172.21.15.90:0/4128144584 conn(0x7f0d2c0b1c80 msgr2=0x7f0d2c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- 172.21.15.90:0/4128144584 shutdown_connections 2024-03-31T15:57:03.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- 172.21.15.90:0/4128144584 wait complete. 2024-03-31T15:57:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 Processor -- start 2024-03-31T15:57:03.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.439+0000 7f0d30e00640 1 -- start start 2024-03-31T15:57:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c06b820 0x7f0d2c1fb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c0cc9d0 0x7f0d2c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c0cf0c0 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c0cef40 con 0x7f0d2c06b820 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2c15c620 con 0x7f0d2c0cc9d0 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49310/0 (socket says 172.21.15.90:49310) 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 -- 172.21.15.90:0/2420045666 learned_addr learned my addr 172.21.15.90:0/2420045666 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d29400640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c0cc9d0 0x7f0d2c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c06b820 0x7f0d2c1fb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 -- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c0cc9d0 msgr2=0x7f0d2c152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c0cc9d0 0x7f0d2c152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 -- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c06b820 msgr2=0x7f0d2c1fb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c06b820 0x7f0d2c1fb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 -- 172.21.15.90:0/2420045666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c1fb870 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d2a800640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400efc0 tx=0x7f0d2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24018020 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24014480 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2c1f7e80 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24014870 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2c1fab50 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2c0686d0 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.443+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d2401b020 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.447+0000 7f0d22000640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 0x7f0d0c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.447+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f0d24087330 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.447+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 0x7f0d0c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.443 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.447+0000 7f0d29e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 0x7f0d0c06a840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c000f70 tx=0x7f0d1c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.451+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f0d24022070 con 0x7f0d2c0ccdb0 2024-03-31T15:57:03.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.603+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d2c167d90 con 0x7f0d0c068380 2024-03-31T15:57:03.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.603+0000 7f0d22000640 1 -- 172.21.15.90:0/2420045666 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13491 (secure 0 0 0) 0x7f0d2c167d90 con 0x7f0d0c068380 2024-03-31T15:57:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:03.599 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:57:03.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 msgr2=0x7f0d0c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 0x7f0d0c06a840 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c000f70 tx=0x7f0d1c007e90 comp rx=0 tx=0).stop 2024-03-31T15:57:03.601 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 msgr2=0x7f0d2c1f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400efc0 tx=0x7f0d2400c490 comp rx=0 tx=0).stop 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 shutdown_connections 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0d0c068380 0x7f0d0c06a840 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0d2c0ccdb0 0x7f0d2c1f7330 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f0d2c0cc9d0 0x7f0d2c152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 --2- 172.21.15.90:0/2420045666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d2c06b820 0x7f0d2c1fb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 >> 172.21.15.90:0/2420045666 conn(0x7f0d2c0b1c80 msgr2=0x7f0d2c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:03.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 shutdown_connections 2024-03-31T15:57:03.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.607+0000 7f0d30e00640 1 -- 172.21.15.90:0/2420045666 wait complete. 2024-03-31T15:57:03.619 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-03-31T15:57:03.263918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222608,"kb_used_data":6448,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752112,"statfs":{"total":773094113280,"available":772866162688,"internally_reserved":0,"allocated":6602752,"data_stored":3619487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":241,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:56:57.800046+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:57.800046+0000","last_peered":"2024-03-31T15:56:57.800046+0000","last_clean":"2024-03-31T15:56:57.800046+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:57.800046+0000","last_undegraded":"2024-03-31T15:56:57.800046+0000","last_fullsized":"2024-03-31T15:56:57.800046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509484,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509485,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":897576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1265664,"data_stored":891341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:57:03.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:57:03.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.759+0000 7f5687800640 1 Processor -- start 2024-03-31T15:57:03.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.759+0000 7f5687800640 1 -- start start 2024-03-31T15:57:03.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5687800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f568015bfd0 0x7f568006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5687800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800b47e0 con 0x7f56801449a0 2024-03-31T15:57:03.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5687800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56800b4ac0 con 0x7f568006bd60 2024-03-31T15:57:03.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5687800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f568006c190 con 0x7f568015bfd0 2024-03-31T15:57:03.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5677e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f568015bfd0 0x7f568006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5684e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f56801449a0 0x7f56800b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50004/0 (socket says 172.21.15.90:50004) 2024-03-31T15:57:03.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5684e00640 1 -- 172.21.15.90:0/2063759282 learned_addr learned my addr 172.21.15.90:0/2063759282 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:03.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5677e00640 1 -- 172.21.15.90:0/2063759282 >> v1:172.21.15.102:6789/0 conn(0x7f568006bd60 legacy=0x7f568015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5677e00640 1 -- 172.21.15.90:0/2063759282 >> v1:172.21.15.90:6789/0 conn(0x7f56801449a0 legacy=0x7f56800b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f5677e00640 1 -- 172.21.15.90:0/2063759282 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568015b7d0 con 0x7f568015bfd0 2024-03-31T15:57:03.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.763+0000 7f567fe00640 1 -- 172.21.15.90:0/2063759282 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905822664 0 0) 0x7f56800b47e0 con 0x7f56801449a0 2024-03-31T15:57:03.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5677e00640 1 --2- 172.21.15.90:0/2063759282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f568006b820 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f566800ba60 tx=0x7f566801eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb1aa80a99e6f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f567fe00640 1 -- 172.21.15.90:0/2063759282 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680b4070 con 0x7f568015bfd0 2024-03-31T15:57:03.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f567fe00640 1 -- 172.21.15.90:0/2063759282 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56680a7ac0 con 0x7f568015bfd0 2024-03-31T15:57:03.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f567fe00640 1 -- 172.21.15.90:0/2063759282 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680af440 con 0x7f568015bfd0 2024-03-31T15:57:03.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- 172.21.15.90:0/2063759282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 msgr2=0x7f568006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 --2- 172.21.15.90:0/2063759282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f568006b820 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f566800ba60 tx=0x7f566801eb70 comp rx=0 tx=0).stop 2024-03-31T15:57:03.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- 172.21.15.90:0/2063759282 shutdown_connections 2024-03-31T15:57:03.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 --2- 172.21.15.90:0/2063759282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f568006b820 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- 172.21.15.90:0/2063759282 >> 172.21.15.90:0/2063759282 conn(0x7f56800b1c80 msgr2=0x7f56800b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:03.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- 172.21.15.90:0/2063759282 shutdown_connections 2024-03-31T15:57:03.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- 172.21.15.90:0/2063759282 wait complete. 2024-03-31T15:57:03.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 Processor -- start 2024-03-31T15:57:03.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- start start 2024-03-31T15:57:03.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f568006bd60 0x7f568015a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f56801449a0 0x7f568013ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f56800b2c60 con 0x7f568015bfd0 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f568015b450 con 0x7f568006bd60 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5687800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f56800b2de0 con 0x7f56801449a0 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.767+0000 7f5685800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49322/0 (socket says 172.21.15.90:49322) 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 -- 172.21.15.90:0/4169950408 learned_addr learned my addr 172.21.15.90:0/4169950408 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:03.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5677e00640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f56801449a0 0x7f568013ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5684e00640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f568006bd60 0x7f568015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 -- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f56801449a0 msgr2=0x7f568013ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f56801449a0 0x7f568013ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 -- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f568006bd60 msgr2=0x7f568015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f568006bd60 0x7f568015a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 -- 172.21.15.90:0/4169950408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568013c320 con 0x7f568015bfd0 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5685800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f567800c7a0 tx=0x7f567800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678003b80 con 0x7f568015bfd0 2024-03-31T15:57:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5678003d20 con 0x7f568015bfd0 2024-03-31T15:57:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567800aa90 con 0x7f568015bfd0 2024-03-31T15:57:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568013c600 con 0x7f568015bfd0 2024-03-31T15:57:03.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568013cbc0 con 0x7f568015bfd0 2024-03-31T15:57:03.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56800686d0 con 0x7f568015bfd0 2024-03-31T15:57:03.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.771+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f567800ac30 con 0x7f568015bfd0 2024-03-31T15:57:03.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.775+0000 7f567d600640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 0x7f565c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:03.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.775+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f5678086690 con 0x7f568015bfd0 2024-03-31T15:57:03.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.775+0000 7f5684e00640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 0x7f565c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:03.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.775+0000 7f5684e00640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 0x7f565c06a870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5670007a90 tx=0x7f56700079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:03.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.779+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f56780566d0 con 0x7f568015bfd0 2024-03-31T15:57:03.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.931+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5680072050 con 0x7f565c0683b0 2024-03-31T15:57:03.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.931+0000 7f567d600640 1 -- 172.21.15.90:0/4169950408 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13491 (secure 0 0 0) 0x7f5680072050 con 0x7f565c0683b0 2024-03-31T15:57:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:03.928 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 msgr2=0x7f565c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 0x7f565c06a870 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5670007a90 tx=0x7f56700079e0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 msgr2=0x7f56800ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f567800c7a0 tx=0x7f567800cc60 comp rx=0 tx=0).stop 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 shutdown_connections 2024-03-31T15:57:03.930 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f565c0683b0 0x7f565c06a870 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f568015bfd0 0x7f56800ce6d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f56801449a0 0x7f568013ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 --2- 172.21.15.90:0/4169950408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f568006bd60 0x7f568015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 >> 172.21.15.90:0/4169950408 conn(0x7f56800b1c80 msgr2=0x7f56800b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 shutdown_connections 2024-03-31T15:57:03.931 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:03.935+0000 7f5687800640 1 -- 172.21.15.90:0/4169950408 wait complete. 2024-03-31T15:57:03.948 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-03-31T15:57:03.263918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222608,"kb_used_data":6448,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752112,"statfs":{"total":773094113280,"available":772866162688,"internally_reserved":0,"allocated":6602752,"data_stored":3619487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":241,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:56:57.800046+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:57.800046+0000","last_peered":"2024-03-31T15:56:57.800046+0000","last_clean":"2024-03-31T15:56:57.800046+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:57.800046+0000","last_undegraded":"2024-03-31T15:56:57.800046+0000","last_fullsized":"2024-03-31T15:56:57.800046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509484,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509485,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":897576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1265664,"data_stored":891341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:57:03.949 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-31T15:57:03.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:57:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.087+0000 7f93d5000640 1 Processor -- start 2024-03-31T15:57:04.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.087+0000 7f93d5000640 1 -- start start 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93d5000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93d015bfd0 0x7f93d006b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93d5000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f93d006bd60 0x7f93d015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93d5000640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d00b47e0 con 0x7f93d01449a0 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93d5000640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f93d006c190 con 0x7f93d006bd60 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93d5000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f93d00b4ac0 con 0x7f93d015bfd0 2024-03-31T15:57:04.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93cde00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f93d01449a0 0x7f93d00b3d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50008/0 (socket says 172.21.15.90:50008) 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93cde00640 1 -- 172.21.15.90:0/1048795009 learned_addr learned my addr 172.21.15.90:0/1048795009 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93ce800640 1 --2- 172.21.15.90:0/1048795009 >> v2:172.21.15.90:3300/0 conn(0x7f93d006bd60 0x7f93d015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93c5400640 1 --2- 172.21.15.90:0/1048795009 >> v2:172.21.15.102:3300/0 conn(0x7f93d015bfd0 0x7f93d006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93cd400640 1 -- 172.21.15.90:0/1048795009 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077532337 0 0) 0x7f93d00b47e0 con 0x7f93d01449a0 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93cd400640 1 -- 172.21.15.90:0/1048795009 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93b4003530 con 0x7f93d01449a0 2024-03-31T15:57:04.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93ce800640 1 -- 172.21.15.90:0/1048795009 >> v2:172.21.15.102:3300/0 conn(0x7f93d015bfd0 msgr2=0x7f93d006b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93ce800640 1 --2- 172.21.15.90:0/1048795009 >> v2:172.21.15.102:3300/0 conn(0x7f93d015bfd0 0x7f93d006b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93ce800640 1 -- 172.21.15.90:0/1048795009 >> v1:172.21.15.90:6789/0 conn(0x7f93d01449a0 legacy=0x7f93d00b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.091+0000 7f93ce800640 1 -- 172.21.15.90:0/1048795009 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d015b7d0 con 0x7f93d006bd60 2024-03-31T15:57:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93ce800640 1 --2- 172.21.15.90:0/1048795009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d006bd60 0x7f93d015ea50 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b1810 tx=0x7f93c801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a652608c3bd62f server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93cd400640 1 -- 172.21.15.90:0/1048795009 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b2070 con 0x7f93d006bd60 2024-03-31T15:57:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93cd400640 1 -- 172.21.15.90:0/1048795009 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80a5870 con 0x7f93d006bd60 2024-03-31T15:57:04.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93cd400640 1 -- 172.21.15.90:0/1048795009 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80a5c40 con 0x7f93d006bd60 2024-03-31T15:57:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- 172.21.15.90:0/1048795009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d006bd60 msgr2=0x7f93d015ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 --2- 172.21.15.90:0/1048795009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d006bd60 0x7f93d015ea50 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f93d00b1810 tx=0x7f93c801c930 comp rx=0 tx=0).stop 2024-03-31T15:57:04.089 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- 172.21.15.90:0/1048795009 shutdown_connections 2024-03-31T15:57:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 --2- 172.21.15.90:0/1048795009 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d006bd60 0x7f93d015ea50 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 --2- 172.21.15.90:0/1048795009 >> v2:172.21.15.102:3300/0 conn(0x7f93d015bfd0 0x7f93d006b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- 172.21.15.90:0/1048795009 >> 172.21.15.90:0/1048795009 conn(0x7f93d00b1c80 msgr2=0x7f93d00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- 172.21.15.90:0/1048795009 shutdown_connections 2024-03-31T15:57:04.090 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- 172.21.15.90:0/1048795009 wait complete. 2024-03-31T15:57:04.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 Processor -- start 2024-03-31T15:57:04.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 -- start start 2024-03-31T15:57:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.095+0000 7f93d5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d006bd60 0x7f93d013f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d015bfd0 0x7f93d0167c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f93d0165de0 con 0x7f93d01449a0 2024-03-31T15:57:04.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f93d015b450 con 0x7f93d006bd60 2024-03-31T15:57:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f93d0165f60 con 0x7f93d015bfd0 2024-03-31T15:57:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49348/0 (socket says 172.21.15.90:49348) 2024-03-31T15:57:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 -- 172.21.15.90:0/4062050960 learned_addr learned my addr 172.21.15.90:0/4062050960 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:04.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93cde00640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d006bd60 0x7f93d013f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93ce800640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d015bfd0 0x7f93d0167c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 -- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d015bfd0 msgr2=0x7f93d0167c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d015bfd0 0x7f93d0167c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 -- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d006bd60 msgr2=0x7f93d013f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d006bd60 0x7f93d013f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 -- 172.21.15.90:0/4062050960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d015e600 con 0x7f93d01449a0 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c5400640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e870 tx=0x7f93b800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b8017070 con 0x7f93d01449a0 2024-03-31T15:57:04.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b800c940 con 0x7f93d01449a0 2024-03-31T15:57:04.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ccf0 con 0x7f93d01449a0 2024-03-31T15:57:04.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d0165360 con 0x7f93d01449a0 2024-03-31T15:57:04.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d0165920 con 0x7f93d01449a0 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d00686d0 con 0x7f93d01449a0 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f93b801a070 con 0x7f93d01449a0 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.099+0000 7f93c6a00640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 0x7f93ac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.103+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f93b8086850 con 0x7f93d01449a0 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.103+0000 7f93cde00640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 0x7f93ac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.103+0000 7f93cde00640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 0x7f93ac06a840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93c0005890 tx=0x7f93c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.107+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f93b8056e50 con 0x7f93d01449a0 2024-03-31T15:57:04.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f93d0072050 con 0x7f93ac068380 2024-03-31T15:57:04.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93c6a00640 1 -- 172.21.15.90:0/4062050960 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13491 (secure 0 0 0) 0x7f93d0072050 con 0x7f93ac068380 2024-03-31T15:57:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:04.258 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 msgr2=0x7f93ac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 0x7f93ac06a840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93c0005890 tx=0x7f93c0009040 comp rx=0 tx=0).stop 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 msgr2=0x7f93d01f7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.263+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e870 tx=0x7f93b800ec70 comp rx=0 tx=0).stop 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 shutdown_connections 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f93ac068380 0x7f93ac06a840 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.260 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f93d015bfd0 0x7f93d0167c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f93d01449a0 0x7f93d01f7360 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 --2- 172.21.15.90:0/4062050960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93d006bd60 0x7f93d013f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 >> 172.21.15.90:0/4062050960 conn(0x7f93d00b1c80 msgr2=0x7f93d00b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:04.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 shutdown_connections 2024-03-31T15:57:04.261 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.267+0000 7f93d5000640 1 -- 172.21.15.90:0/4062050960 wait complete. 2024-03-31T15:57:04.278 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-03-31T15:57:03.263918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":222608,"kb_used_data":6448,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752112,"statfs":{"total":773094113280,"available":772866162688,"internally_reserved":0,"allocated":6602752,"data_stored":3619487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":241,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-31T15:56:57.800046+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:56:57.800046+0000","last_peered":"2024-03-31T15:56:57.800046+0000","last_clean":"2024-03-31T15:56:57.800046+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:56:57.800046+0000","last_undegraded":"2024-03-31T15:56:57.800046+0000","last_fullsized":"2024-03-31T15:56:57.800046+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509484,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509484,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509485,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":307208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509485,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1273856,"data_stored":897576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":300869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343584,"statfs":{"total":96636764160,"available":96607830016,"internally_reserved":0,"allocated":1265664,"data_stored":891341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:57:04.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-31T15:57:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 Processor -- start 2024-03-31T15:57:04.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 -- start start 2024-03-31T15:57:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbab40cd810 0x7fbab415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab40cdc80 con 0x7fbab40b1760 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab40b51f0 con 0x7fbab40ccd20 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbab40b5e00 con 0x7fbab40cd810 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbaaa000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbab40b1760 0x7fbab415d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50016/0 (socket says 172.21.15.90:50016) 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbaaa000640 1 -- 172.21.15.90:0/1598165697 learned_addr learned my addr 172.21.15.90:0/1598165697 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197876868 0 0) 0x7fba9c001120 con 0x7fbab40b1760 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3400640 1 --2- 172.21.15.90:0/1598165697 >> v2:172.21.15.102:3300/0 conn(0x7fbab40cd810 0x7fbab415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba98003530 con 0x7fbab40b1760 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2269257241 0 0) 0x7fba98003530 con 0x7fbab40b1760 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 >> v2:172.21.15.102:3300/0 conn(0x7fbab40cd810 msgr2=0x7fbab415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 --2- 172.21.15.90:0/1598165697 >> v2:172.21.15.102:3300/0 conn(0x7fbab40cd810 0x7fbab415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 >> v1:172.21.15.102:6789/0 conn(0x7fbab40ccd20 legacy=0x7fbab40cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab40b54d0 con 0x7fbab40b1760 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fba9c001850 con 0x7fbab40b1760 2024-03-31T15:57:04.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 >> v1:172.21.15.90:6789/0 conn(0x7fbab40b1760 legacy=0x7fbab415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fba980039c0 0x7fba98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba98004810 0x7fba98004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 0x7fba9800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbab40b5e00 con 0x7fba980051d0 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba9800c020 con 0x7fba98004810 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fba9800c220 con 0x7fba980039c0 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab3400640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 0x7fba9800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbab2a00640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fba980039c0 0x7fba98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.419+0000 7fbaaa000640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba98004810 0x7fba98004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 -- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fba980039c0 msgr2=0x7fba98003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fba980039c0 0x7fba98003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 -- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba98004810 msgr2=0x7fba98004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba98004810 0x7fba98004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 -- 172.21.15.90:0/1598165697 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba98003530 con 0x7fba980051d0 2024-03-31T15:57:04.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 -- 172.21.15.90:0/1598165697 reap_dead start 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3400640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 0x7fba9800b9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009fc0 tx=0x7fbaac01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbc638aeb675278 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac009c10 con 0x7fba980051d0 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaac0a5bc0 con 0x7fba980051d0 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab2000640 1 -- 172.21.15.90:0/1598165697 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0039a0 con 0x7fba980051d0 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3e00640 1 -- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 msgr2=0x7fba9800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 0x7fba9800b9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009fc0 tx=0x7fbaac01ccc0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3e00640 1 -- 172.21.15.90:0/1598165697 shutdown_connections 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1598165697 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fba980051d0 0x7fba9800b9d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.423+0000 7fbab3e00640 1 -- 172.21.15.90:0/1598165697 >> 172.21.15.90:0/1598165697 conn(0x7fbab4072d80 msgr2=0x7fbab4073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- 172.21.15.90:0/1598165697 shutdown_connections 2024-03-31T15:57:04.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- 172.21.15.90:0/1598165697 wait complete. 2024-03-31T15:57:04.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 Processor -- start 2024-03-31T15:57:04.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- start start 2024-03-31T15:57:04.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba980051d0 0x7fbab4160220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbab409b470 0x7fbab409a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbab40b5a90 con 0x7fbab409dfa0 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbab40b5050 con 0x7fba980051d0 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbab40b6290 con 0x7fbab409b470 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49370/0 (socket says 172.21.15.90:49370) 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 -- 172.21.15.90:0/1770085603 learned_addr learned my addr 172.21.15.90:0/1770085603 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab2a00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba980051d0 0x7fbab4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3400640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbab409b470 0x7fbab409a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 -- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbab409b470 msgr2=0x7fbab409a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbab409b470 0x7fbab409a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 -- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba980051d0 msgr2=0x7fbab4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba980051d0 0x7fbab4160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 -- 172.21.15.90:0/1770085603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab409cf60 con 0x7fbab409dfa0 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaaa000640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0019c0 tx=0x7fba9c012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c01c070 con 0x7fbab409dfa0 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c0155e0 con 0x7fbab409dfa0 2024-03-31T15:57:04.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c0159d0 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab409baa0 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.427+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab409d4c0 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab4068d00 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba9c01f070 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbaab400640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 0x7fba9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fba9c088ff0 con 0x7fbab409dfa0 2024-03-31T15:57:04.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbab2a00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 0x7fba9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:04.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.431+0000 7fbab2a00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 0x7fba9006a840 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbab406bc60 tx=0x7fbaa4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:04.431 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.435+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fbab409cf60 con 0x7fbab409dfa0 2024-03-31T15:57:04.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.591+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbab4160760 con 0x7fbab409dfa0 2024-03-31T15:57:04.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.591+0000 7fbaab400640 1 -- 172.21.15.90:0/1770085603 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+13257 (secure 0 0 0) 0x7fbab40cdc80 con 0x7fbab409dfa0 2024-03-31T15:57:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:04.588 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":58,"fsid":"0135ca30-606a-448f-b61f-3ca22bf9710a","created":"2024-03-31T15:53:24.176166+0000","modified":"2024-03-31T15:57:01.162940+0000","last_up_change":"2024-03-31T15:53:35.674647+0000","last_in_change":"2024-03-31T15:53:26.636888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-31T15:53:35.236139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c67e90a6-0b88-4027-a912-d44a80e48d1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6811","nonce":2322588474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6817","nonce":2322588474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6823","nonce":2322588474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":2322588474},{"type":"v1","addr":"172.21.15.90:6821","nonce":2322588474}]},"public_addr":"172.21.15.90:6811/2322588474","cluster_addr":"172.21.15.90:6817/2322588474","heartbeat_back_addr":"172.21.15.90:6823/2322588474","heartbeat_front_addr":"172.21.15.90:6821/2322588474","state":["exists","up"]},{"osd":1,"uuid":"e719019d-e1d1-415e-b65f-56fd94db3bae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6825","nonce":3383585142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6827","nonce":3383585142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6831","nonce":3383585142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":3383585142},{"type":"v1","addr":"172.21.15.90:6829","nonce":3383585142}]},"public_addr":"172.21.15.90:6825/3383585142","cluster_addr":"172.21.15.90:6827/3383585142","heartbeat_back_addr":"172.21.15.90:6831/3383585142","heartbeat_front_addr":"172.21.15.90:6829/3383585142","state":["exists","up"]},{"osd":2,"uuid":"1f39c8a8-92d5-4c1e-ba1a-68be683b063c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6801","nonce":347246527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6803","nonce":347246527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6807","nonce":347246527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":347246527},{"type":"v1","addr":"172.21.15.90:6805","nonce":347246527}]},"public_addr":"172.21.15.90:6801/347246527","cluster_addr":"172.21.15.90:6803/347246527","heartbeat_back_addr":"172.21.15.90:6807/347246527","heartbeat_front_addr":"172.21.15.90:6805/347246527","state":["exists","up"]},{"osd":3,"uuid":"ff0b59be-f0c8-44ea-82f1-5b644069371b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6810","nonce":4045683431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6813","nonce":4045683431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6820","nonce":4045683431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6815","nonce":4045683431},{"type":"v1","addr":"172.21.15.90:6816","nonce":4045683431}]},"public_addr":"172.21.15.90:6810/4045683431","cluster_addr":"172.21.15.90:6813/4045683431","heartbeat_back_addr":"172.21.15.90:6820/4045683431","heartbeat_front_addr":"172.21.15.90:6816/4045683431","state":["exists","up"]},{"osd":4,"uuid":"5305284e-adc1-4c1f-882b-75769fe3ad3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6801","nonce":101321808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6803","nonce":101321808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6807","nonce":101321808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":101321808},{"type":"v1","addr":"172.21.15.102:6805","nonce":101321808}]},"public_addr":"172.21.15.102:6801/101321808","cluster_addr":"172.21.15.102:6803/101321808","heartbeat_back_addr":"172.21.15.102:6807/101321808","heartbeat_front_addr":"172.21.15.102:6805/101321808","state":["exists","up"]},{"osd":5,"uuid":"31d64299-ceb7-4ab7-b692-45a8b7bf01df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6810","nonce":3785791459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6814","nonce":3785791459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6822","nonce":3785791459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3785791459},{"type":"v1","addr":"172.21.15.102:6818","nonce":3785791459}]},"public_addr":"172.21.15.102:6810/3785791459","cluster_addr":"172.21.15.102:6814/3785791459","heartbeat_back_addr":"172.21.15.102:6822/3785791459","heartbeat_front_addr":"172.21.15.102:6818/3785791459","state":["exists","up"]},{"osd":6,"uuid":"4ee5275f-1b33-4e41-8337-316a3de87dfd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6825","nonce":948959501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6827","nonce":948959501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6831","nonce":948959501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":948959501},{"type":"v1","addr":"172.21.15.102:6829","nonce":948959501}]},"public_addr":"172.21.15.102:6825/948959501","cluster_addr":"172.21.15.102:6827/948959501","heartbeat_back_addr":"172.21.15.102:6831/948959501","heartbeat_front_addr":"172.21.15.102:6829/948959501","state":["exists","up"]},{"osd":7,"uuid":"636531f9-c5c4-4648-92b5-f3747e70afa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6811","nonce":552876268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6815","nonce":552876268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6823","nonce":552876268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":552876268},{"type":"v1","addr":"172.21.15.102:6819","nonce":552876268}]},"public_addr":"172.21.15.102:6811/552876268","cluster_addr":"172.21.15.102:6815/552876268","heartbeat_back_addr":"172.21.15.102:6823/552876268","heartbeat_front_addr":"172.21.15.102:6819/552876268","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.536481+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.564783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.138532+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:33.727099+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.267580+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.271153+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.270359+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-31T15:53:32.067051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6835/1583024065":"2024-04-01T15:56:57.792540+0000","172.21.15.90:6833/1583024065":"2024-04-01T15:56:57.792540+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 msgr2=0x7fba9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 0x7fba9006a840 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbab406bc60 tx=0x7fbaa4009040 comp rx=0 tx=0).stop 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 msgr2=0x7fbab409af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0019c0 tx=0x7fba9c012630 comp rx=0 tx=0).stop 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 shutdown_connections 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fba90068380 0x7fba9006a840 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbab409b470 0x7fbab409a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbab409dfa0 0x7fbab409af30 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 --2- 172.21.15.90:0/1770085603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba980051d0 0x7fbab4160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 >> 172.21.15.90:0/1770085603 conn(0x7fbab4072d80 msgr2=0x7fbab40ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 shutdown_connections 2024-03-31T15:57:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:04.595+0000 7fbab3e00640 1 -- 172.21.15.90:0/1770085603 wait complete. 2024-03-31T15:57:05.609 INFO:tasks.ceph:Scrubbing osd.0 2024-03-31T15:57:05.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 Processor -- start 2024-03-31T15:57:05.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 -- start start 2024-03-31T15:57:05.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd3806b850 0x7fbd380b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd3806bc30 0x7fbd380b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd38161e30 con 0x7fbd380d0b80 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fbd38162110 con 0x7fbd3806b850 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd3e800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbd381623f0 con 0x7fbd3806bc30 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fbd3806b850 0x7fbd380b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbd380d0b80 0x7fbd380d0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50028/0 (socket says 172.21.15.90:50028) 2024-03-31T15:57:05.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37e00640 1 -- 172.21.15.90:0/1876805086 learned_addr learned my addr 172.21.15.90:0/1876805086 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37400640 1 -- 172.21.15.90:0/1876805086 >> v2:172.21.15.102:3300/0 conn(0x7fbd3806bc30 msgr2=0x7fbd380b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37400640 1 --2- 172.21.15.90:0/1876805086 >> v2:172.21.15.102:3300/0 conn(0x7fbd3806bc30 0x7fbd380b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37400640 1 -- 172.21.15.90:0/1876805086 >> v1:172.21.15.90:6789/0 conn(0x7fbd380d0b80 legacy=0x7fbd380d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd37400640 1 -- 172.21.15.90:0/1876805086 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381626d0 con 0x7fbd3806b850 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.763+0000 7fbd36000640 1 -- 172.21.15.90:0/1876805086 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892227012 0 0) 0x7fbd38161e30 con 0x7fbd380d0b80 2024-03-31T15:57:05.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd37400640 1 --2- 172.21.15.90:0/1876805086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd3806b850 0x7fbd380b5820 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbd380722c0 tx=0x7fbd2801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca12ab1ff702ef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd36000640 1 -- 172.21.15.90:0/1876805086 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd280b0070 con 0x7fbd3806b850 2024-03-31T15:57:05.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd36000640 1 -- 172.21.15.90:0/1876805086 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd280ab600 con 0x7fbd3806b850 2024-03-31T15:57:05.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd36000640 1 -- 172.21.15.90:0/1876805086 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd280ab9d0 con 0x7fbd3806b850 2024-03-31T15:57:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 -- 172.21.15.90:0/1876805086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd3806b850 msgr2=0x7fbd380b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 --2- 172.21.15.90:0/1876805086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd3806b850 0x7fbd380b5820 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbd380722c0 tx=0x7fbd2801c480 comp rx=0 tx=0).stop 2024-03-31T15:57:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 -- 172.21.15.90:0/1876805086 shutdown_connections 2024-03-31T15:57:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 --2- 172.21.15.90:0/1876805086 >> v2:172.21.15.102:3300/0 conn(0x7fbd3806bc30 0x7fbd380b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 --2- 172.21.15.90:0/1876805086 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd3806b850 0x7fbd380b5820 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 -- 172.21.15.90:0/1876805086 >> 172.21.15.90:0/1876805086 conn(0x7fbd3806c700 msgr2=0x7fbd3806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 -- 172.21.15.90:0/1876805086 shutdown_connections 2024-03-31T15:57:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.767+0000 7fbd3e800640 1 -- 172.21.15.90:0/1876805086 wait complete. 2024-03-31T15:57:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 Processor -- start 2024-03-31T15:57:05.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- start start 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3806b850 0x7fbd381f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd3806bc30 0x7fbd380b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 0x7fbd380af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380d1c30 con 0x7fbd380d0b80 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd380d1ab0 con 0x7fbd3806b850 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3816b130 con 0x7fbd3806bc30 2024-03-31T15:57:05.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 0x7fbd380af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 0x7fbd380af4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49378/0 (socket says 172.21.15.90:49378) 2024-03-31T15:57:05.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 -- 172.21.15.90:0/795056897 learned_addr learned my addr 172.21.15.90:0/795056897 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:05.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37400640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3806b850 0x7fbd381f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd36a00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd3806bc30 0x7fbd380b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 -- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd3806bc30 msgr2=0x7fbd380b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fbd3806bc30 0x7fbd380b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 -- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3806b850 msgr2=0x7fbd381f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3806b850 0x7fbd381f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd381f7e10 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd37e00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 0x7fbd380af4d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000c7a0 tx=0x7fbd3000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30018070 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3000a640 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd381f80f0 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000aab0 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.771+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd381f86b0 con 0x7fbd380d0b80 2024-03-31T15:57:05.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd380686d0 con 0x7fbd380d0b80 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbd3001b070 con 0x7fbd380d0b80 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbd180683b0 0x7fbd1806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fbd300863d0 con 0x7fbd380d0b80 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7fbd1806e850 0x7fbd18070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd18071360 con 0x7fbd1806e850 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fbd30086780 con 0x7fbd380d0b80 2024-03-31T15:57:05.773 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd37400640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbd180683b0 0x7fbd1806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd36a00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7fbd1806e850 0x7fbd18070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd36a00640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7fbd1806e850 0x7fbd18070c90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd37400640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbd180683b0 0x7fbd1806a870 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbd280ad000 tx=0x7fbd280a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.774 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.775+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== osd.0 v2:172.21.15.90:6809/2322588474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd18071360 con 0x7fbd1806e850 2024-03-31T15:57:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 --> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbd38072b80 con 0x7fbd1806e850 2024-03-31T15:57:05.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd2f400640 1 -- 172.21.15.90:0/795056897 <== osd.0 v2:172.21.15.90:6809/2322588474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbd38072b80 con 0x7fbd1806e850 2024-03-31T15:57:05.785 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:05.785 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:05.785 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7fbd1806e850 msgr2=0x7fbd18070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:6809/2322588474,v1:172.21.15.90:6811/2322588474] conn(0x7fbd1806e850 0x7fbd18070c90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbd180683b0 msgr2=0x7fbd1806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fbd180683b0 0x7fbd1806a870 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbd280ad000 tx=0x7fbd280a4f90 comp rx=0 tx=0).stop 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 msgr2=0x7fbd380af4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 --2- 172.21.15.90:0/795056897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbd380d0b80 0x7fbd380af4d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000c7a0 tx=0x7fbd3000cc60 comp rx=0 tx=0).stop 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd37e00640 1 -- 172.21.15.90:0/795056897 reap_dead start 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 shutdown_connections 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 >> 172.21.15.90:0/795056897 conn(0x7fbd3806c700 msgr2=0x7fbd380ae0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 shutdown_connections 2024-03-31T15:57:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.791+0000 7fbd3e800640 1 -- 172.21.15.90:0/795056897 wait complete. 2024-03-31T15:57:05.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-31T15:57:05.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.943+0000 7f11f4600640 1 Processor -- start 2024-03-31T15:57:05.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.943+0000 7f11f4600640 1 -- start start 2024-03-31T15:57:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f4600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11ec151d70 0x7f11ec0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f4600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec152150 0x7f11ec15f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f4600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec071cd0 con 0x7f11ec072a90 2024-03-31T15:57:05.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f4600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec071fb0 con 0x7f11ec151d70 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f4600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec072290 con 0x7f11ec152150 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f11ec072a90 0x7f11ec15efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50040/0 (socket says 172.21.15.90:50040) 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1200640 1 -- 172.21.15.90:0/1683217633 learned_addr learned my addr 172.21.15.90:0/1683217633 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1c00640 1 --2- 172.21.15.90:0/1683217633 >> v2:172.21.15.102:3300/0 conn(0x7f11ec151d70 0x7f11ec0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f2600640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec152150 0x7f11ec15f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11ebe00640 1 -- 172.21.15.90:0/1683217633 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2641640134 0 0) 0x7f11ec071cd0 con 0x7f11ec072a90 2024-03-31T15:57:05.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11ebe00640 1 -- 172.21.15.90:0/1683217633 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d4003530 con 0x7f11ec072a90 2024-03-31T15:57:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1c00640 1 -- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec152150 msgr2=0x7f11ec15f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1c00640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec152150 0x7f11ec15f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1c00640 1 -- 172.21.15.90:0/1683217633 >> v1:172.21.15.90:6789/0 conn(0x7f11ec072a90 legacy=0x7f11ec15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.947+0000 7f11f1c00640 1 -- 172.21.15.90:0/1683217633 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec072570 con 0x7f11ec151d70 2024-03-31T15:57:05.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f1c00640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec151d70 0x7f11ec0cc440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f11e00097c0 tx=0x7f11e001cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be0a8ab15407bbef server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11ebe00640 1 -- 172.21.15.90:0/1683217633 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00aa620 con 0x7f11ec151d70 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11ebe00640 1 -- 172.21.15.90:0/1683217633 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e00af070 con 0x7f11ec151d70 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11ebe00640 1 -- 172.21.15.90:0/1683217633 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00aa9c0 con 0x7f11ec151d70 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec151d70 msgr2=0x7f11ec0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec151d70 0x7f11ec0cc440 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f11e00097c0 tx=0x7f11e001cf00 comp rx=0 tx=0).stop 2024-03-31T15:57:05.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- 172.21.15.90:0/1683217633 shutdown_connections 2024-03-31T15:57:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec152150 0x7f11ec15f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 --2- 172.21.15.90:0/1683217633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec151d70 0x7f11ec0cc440 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- 172.21.15.90:0/1683217633 >> 172.21.15.90:0/1683217633 conn(0x7f11ec0b1a50 msgr2=0x7f11ec0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- 172.21.15.90:0/1683217633 shutdown_connections 2024-03-31T15:57:05.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- 172.21.15.90:0/1683217633 wait complete. 2024-03-31T15:57:05.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 Processor -- start 2024-03-31T15:57:05.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.951+0000 7f11f4600640 1 -- start start 2024-03-31T15:57:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec072a90 0x7f11ec0b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec151d70 0x7f11ec0cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec0ca380 con 0x7f11ec152150 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec15c330 con 0x7f11ec072a90 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec0ca500 con 0x7f11ec151d70 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49382/0 (socket says 172.21.15.90:49382) 2024-03-31T15:57:05.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 -- 172.21.15.90:0/3378520842 learned_addr learned my addr 172.21.15.90:0/3378520842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f1200640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec151d70 0x7f11ec0cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f1c00640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec072a90 0x7f11ec0b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 -- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec151d70 msgr2=0x7f11ec0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec151d70 0x7f11ec0cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 -- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec072a90 msgr2=0x7f11ec0b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec072a90 0x7f11ec0b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:05.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 -- 172.21.15.90:0/3378520842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec155c80 con 0x7f11ec152150 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f2600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f11e400efc0 tx=0x7f11e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4018020 con 0x7f11ec152150 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e4014480 con 0x7f11ec152150 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4014870 con 0x7f11ec152150 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec155f20 con 0x7f11ec152150 2024-03-31T15:57:05.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec0b4eb0 con 0x7f11ec152150 2024-03-31T15:57:05.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec068660 con 0x7f11ec152150 2024-03-31T15:57:05.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.955+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f11e401b020 con 0x7f11ec152150 2024-03-31T15:57:05.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.959+0000 7f11e9600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 0x7f11cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:05.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.959+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f11e40871c0 con 0x7f11ec152150 2024-03-31T15:57:05.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.959+0000 7f11f1c00640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 0x7f11cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:05.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.959+0000 7f11f1c00640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 0x7f11cc06a840 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f11e00ae000 tx=0x7f11e0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:05.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:05.963+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f11e4022070 con 0x7f11ec152150 2024-03-31T15:57:06.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.119+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f11ec0cdee0 con 0x7f11cc068380 2024-03-31T15:57:06.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.119+0000 7f11e9600640 1 -- 172.21.15.90:0/3378520842 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f11ec0cdee0 con 0x7f11cc068380 2024-03-31T15:57:06.114 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 0 to deep-scrub 2024-03-31T15:57:06.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 msgr2=0x7f11cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 0x7f11cc06a840 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f11e00ae000 tx=0x7f11e0004740 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 msgr2=0x7f11ec157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f11e400efc0 tx=0x7f11e400c490 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 shutdown_connections 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f11cc068380 0x7f11cc06a840 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11ec152150 0x7f11ec157510 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f11ec151d70 0x7f11ec0cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 --2- 172.21.15.90:0/3378520842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec072a90 0x7f11ec0b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 >> 172.21.15.90:0/3378520842 conn(0x7f11ec0b1a50 msgr2=0x7f11ec0cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 shutdown_connections 2024-03-31T15:57:06.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.123+0000 7f11f4600640 1 -- 172.21.15.90:0/3378520842 wait complete. 2024-03-31T15:57:06.135 INFO:tasks.ceph:Scrubbing osd.1 2024-03-31T15:57:06.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:06.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 Processor -- start 2024-03-31T15:57:06.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 -- start start 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38d406eac0 0x7f38d415eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d415b530 con 0x7f38d415b150 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38d415f310 con 0x7f38d415bce0 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d8c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f38d415f0e0 con 0x7f38d406eac0 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f38d415b150 0x7f38d40ca9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50056/0 (socket says 172.21.15.90:50056) 2024-03-31T15:57:06.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 -- 172.21.15.90:0/1191478711 learned_addr learned my addr 172.21.15.90:0/1191478711 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1400640 1 --2- 172.21.15.90:0/1191478711 >> v2:172.21.15.102:3300/0 conn(0x7f38d406eac0 0x7f38d415eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672362646 0 0) 0x7f38d415b530 con 0x7f38d415b150 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38b8003530 con 0x7f38d415b150 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1227569991 0 0) 0x7f38d415f310 con 0x7f38d415bce0 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38d415b530 con 0x7f38d415bce0 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 725260676 0 0) 0x7f38b8003530 con 0x7f38d415b150 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 >> v2:172.21.15.102:3300/0 conn(0x7f38d406eac0 msgr2=0x7f38d415eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.281 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 --2- 172.21.15.90:0/1191478711 >> v2:172.21.15.102:3300/0 conn(0x7f38d406eac0 0x7f38d415eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 >> v1:172.21.15.102:6789/0 conn(0x7f38d415bce0 legacy=0x7f38d40cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d415f5f0 con 0x7f38d415b150 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f38c4001a00 con 0x7f38d415b150 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 >> v1:172.21.15.90:6789/0 conn(0x7f38d415b150 legacy=0x7f38d40ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 0x7f38b8003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8004810 0x7f38b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38b80051d0 0x7f38b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f38b800c0a0 con 0x7f38b8003ba0 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38b800c220 con 0x7f38b80051d0 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f38b800c420 con 0x7f38b8004810 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 0x7f38b8003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1400640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8004810 0x7f38b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d2800640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38b80051d0 0x7f38b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 -- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8004810 msgr2=0x7f38b8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8004810 0x7f38b8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 -- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38b80051d0 msgr2=0x7f38b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38b80051d0 0x7f38b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d2800640 1 -- 172.21.15.90:0/1191478711 reap_dead start 2024-03-31T15:57:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.283+0000 7f38d1e00640 1 -- 172.21.15.90:0/1191478711 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b8003530 con 0x7f38b8003ba0 2024-03-31T15:57:06.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 0x7f38b8003fa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f38c400a750 tx=0x7f38c401d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d60792efbe6c66e server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.287 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c400a370 con 0x7f38b8003ba0 2024-03-31T15:57:06.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c40ad630 con 0x7f38b8003ba0 2024-03-31T15:57:06.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d0a00640 1 -- 172.21.15.90:0/1191478711 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40ada20 con 0x7f38b8003ba0 2024-03-31T15:57:06.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 -- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 msgr2=0x7f38b8003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 0x7f38b8003fa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f38c400a750 tx=0x7f38c401d260 comp rx=0 tx=0).stop 2024-03-31T15:57:06.288 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 -- 172.21.15.90:0/1191478711 shutdown_connections 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 --2- 172.21.15.90:0/1191478711 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38b8003ba0 0x7f38b8003fa0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 -- 172.21.15.90:0/1191478711 >> 172.21.15.90:0/1191478711 conn(0x7f38d40b1880 msgr2=0x7f38d40b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 -- 172.21.15.90:0/1191478711 shutdown_connections 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.287+0000 7f38d8c00640 1 -- 172.21.15.90:0/1191478711 wait complete. 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 Processor -- start 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- start start 2024-03-31T15:57:06.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8003ba0 0x7f38d4158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 0x7f38d40c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d41f89d0 0x7f38d41f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40cb1f0 con 0x7f38d40c8490 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38d415b850 con 0x7f38d41f89d0 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f38d40cc480 con 0x7f38b8003ba0 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 0x7f38d40c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 0x7f38d40c95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49400/0 (socket says 172.21.15.90:49400) 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 -- 172.21.15.90:0/1578837223 learned_addr learned my addr 172.21.15.90:0/1578837223 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8003ba0 0x7f38d4158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d2800640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d41f89d0 0x7f38d41f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 -- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8003ba0 msgr2=0x7f38d4158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f38b8003ba0 0x7f38d4158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 -- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d41f89d0 msgr2=0x7f38d41f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.291 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d41f89d0 0x7f38d41f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d40c5ee0 con 0x7f38d40c8490 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d2800640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d41f89d0 0x7f38d41f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d1400640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 0x7f38d40c95f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f38c0011500 tx=0x7f38c00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0015870 con 0x7f38d40c8490 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38c0015a10 con 0x7f38d40c8490 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d40c79b0 con 0x7f38d40c8490 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0015dc0 con 0x7f38d40c8490 2024-03-31T15:57:06.292 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.291+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d40c7f70 con 0x7f38d40c8490 2024-03-31T15:57:06.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38d40c77c0 con 0x7f38d40c8490 2024-03-31T15:57:06.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38c0013400 con 0x7f38d40c8490 2024-03-31T15:57:06.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38b40683b0 0x7f38b406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f38c00873f0 con 0x7f38d40c8490 2024-03-31T15:57:06.294 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f38b406e850 0x7f38b4070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38c0027ec0 con 0x7f38b406e850 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f38c00877a0 con 0x7f38d40c8490 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38b40683b0 0x7f38b406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38d2800640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f38b406e850 0x7f38b4070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38d2800640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f38b406e850 0x7f38b4070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.295+0000 7f38d1e00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38b40683b0 0x7f38b406a870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f38c40ad000 tx=0x7f38c40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.295 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.299+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== osd.1 v2:172.21.15.90:6824/3383585142 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38c0027ec0 con 0x7f38b406e850 2024-03-31T15:57:06.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 --> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f38d40686d0 con 0x7f38b406e850 2024-03-31T15:57:06.305 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38ca000640 1 -- 172.21.15.90:0/1578837223 <== osd.1 v2:172.21.15.90:6824/3383585142 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f38d40686d0 con 0x7f38b406e850 2024-03-31T15:57:06.305 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:06.305 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:06.305 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f38b406e850 msgr2=0x7f38b4070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.306 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:6824/3383585142,v1:172.21.15.90:6825/3383585142] conn(0x7f38b406e850 0x7f38b4070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38b40683b0 msgr2=0x7f38b406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f38b40683b0 0x7f38b406a870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f38c40ad000 tx=0x7f38c40a7000 comp rx=0 tx=0).stop 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 msgr2=0x7f38d40c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 --2- 172.21.15.90:0/1578837223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38d40c8490 0x7f38d40c95f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f38c0011500 tx=0x7f38c00119c0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d2800640 1 -- 172.21.15.90:0/1578837223 reap_dead start 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 shutdown_connections 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 >> 172.21.15.90:0/1578837223 conn(0x7f38d40b1880 msgr2=0x7f38d4072490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 shutdown_connections 2024-03-31T15:57:06.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.311+0000 7f38d8c00640 1 -- 172.21.15.90:0/1578837223 wait complete. 2024-03-31T15:57:06.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-31T15:57:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 Processor -- start 2024-03-31T15:57:06.460 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 -- start start 2024-03-31T15:57:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98e4151d70 0x7f98e40cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.461 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4152150 0x7f98e415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e4071cd0 con 0x7f98e4072a90 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f98e4071fb0 con 0x7f98e4151d70 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98ec600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f98e4072290 con 0x7f98e4152150 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f98e4151d70 0x7f98e40cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f98e4072a90 0x7f98e415efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50062/0 (socket says 172.21.15.90:50062) 2024-03-31T15:57:06.462 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9200640 1 -- 172.21.15.90:0/2816920913 learned_addr learned my addr 172.21.15.90:0/2816920913 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9c00640 1 -- 172.21.15.90:0/2816920913 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4152150 msgr2=0x7f98e415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9c00640 1 --2- 172.21.15.90:0/2816920913 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4152150 0x7f98e415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9c00640 1 -- 172.21.15.90:0/2816920913 >> v1:172.21.15.90:6789/0 conn(0x7f98e4072a90 legacy=0x7f98e415efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.463 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.467+0000 7f98e9c00640 1 -- 172.21.15.90:0/2816920913 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e4072570 con 0x7f98e4151d70 2024-03-31T15:57:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98e9c00640 1 --2- 172.21.15.90:0/2816920913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4151d70 0x7f98e40cc440 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f98d80097c0 tx=0x7f98d801cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc77879aa4f45251 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98e3e00640 1 -- 172.21.15.90:0/2816920913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d80aa620 con 0x7f98e4151d70 2024-03-31T15:57:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98e3e00640 1 -- 172.21.15.90:0/2816920913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d80af070 con 0x7f98e4151d70 2024-03-31T15:57:06.465 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98e3e00640 1 -- 172.21.15.90:0/2816920913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d80aa9e0 con 0x7f98e4151d70 2024-03-31T15:57:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 -- 172.21.15.90:0/2816920913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4151d70 msgr2=0x7f98e40cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.466 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 --2- 172.21.15.90:0/2816920913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4151d70 0x7f98e40cc440 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f98d80097c0 tx=0x7f98d801cf00 comp rx=0 tx=0).stop 2024-03-31T15:57:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 -- 172.21.15.90:0/2816920913 shutdown_connections 2024-03-31T15:57:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 --2- 172.21.15.90:0/2816920913 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4152150 0x7f98e415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 --2- 172.21.15.90:0/2816920913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4151d70 0x7f98e40cc440 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.469 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 -- 172.21.15.90:0/2816920913 >> 172.21.15.90:0/2816920913 conn(0x7f98e40b1a50 msgr2=0x7f98e40b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 -- 172.21.15.90:0/2816920913 shutdown_connections 2024-03-31T15:57:06.470 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 -- 172.21.15.90:0/2816920913 wait complete. 2024-03-31T15:57:06.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.471+0000 7f98ec600640 1 Processor -- start 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- start start 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4151d70 0x7f98e40c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98e4152150 0x7f98e4155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f98e40ca380 con 0x7f98e4072a90 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f98e415c330 con 0x7f98e4152150 2024-03-31T15:57:06.474 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f98e40ca500 con 0x7f98e4151d70 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49422/0 (socket says 172.21.15.90:49422) 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 -- 172.21.15.90:0/2076908465 learned_addr learned my addr 172.21.15.90:0/2076908465 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 -- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4151d70 msgr2=0x7f98e40c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4151d70 0x7f98e40c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 -- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98e4152150 msgr2=0x7f98e4155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98e4152150 0x7f98e4155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 -- 172.21.15.90:0/2076908465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e4158200 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e9c00640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f98d801ab50 tx=0x7f98d801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d80aae10 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d80be410 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e41584e0 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d80be800 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e4158aa0 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.475+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e4068660 con 0x7f98e4072a90 2024-03-31T15:57:06.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.479+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98d80bc020 con 0x7f98e4072a90 2024-03-31T15:57:06.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.479+0000 7f98e1600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 0x7f98c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.479+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f98d8120d10 con 0x7f98e4072a90 2024-03-31T15:57:06.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.479+0000 7f98e9200640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 0x7f98c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.479+0000 7f98e9200640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 0x7f98c406a840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000f70 tx=0x7f98d0007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.483+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f98d8126020 con 0x7f98e4072a90 2024-03-31T15:57:06.632 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f98e4154f90 con 0x7f98c4068380 2024-03-31T15:57:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98e1600640 1 -- 172.21.15.90:0/2076908465 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f98d00058c0 con 0x7f98c4068380 2024-03-31T15:57:06.633 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 1 to deep-scrub 2024-03-31T15:57:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 msgr2=0x7f98c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 0x7f98c406a840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f98d0000f70 tx=0x7f98d0007a90 comp rx=0 tx=0).stop 2024-03-31T15:57:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 msgr2=0x7f98e4154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f98d801ab50 tx=0x7f98d801ace0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 shutdown_connections 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f98c4068380 0x7f98c406a840 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98e4152150 0x7f98e4155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f98e4151d70 0x7f98e40c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 --2- 172.21.15.90:0/2076908465 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f98e4072a90 0x7f98e4154a50 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 >> 172.21.15.90:0/2076908465 conn(0x7f98e40b1a50 msgr2=0x7f98e406d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 shutdown_connections 2024-03-31T15:57:06.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.639+0000 7f98ec600640 1 -- 172.21.15.90:0/2076908465 wait complete. 2024-03-31T15:57:06.654 INFO:tasks.ceph:Scrubbing osd.2 2024-03-31T15:57:06.655 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:06.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 Processor -- start 2024-03-31T15:57:06.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 -- start start 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f083c15be10 0x7f083c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c15c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f083c15cd60 con 0x7f083c0cb630 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f083c15fdc0 con 0x7f083c15be10 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f0842400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f083c15d040 con 0x7f083c0cbae0 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.799+0000 7f083b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f083c15be10 0x7f083c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f083c15be10 0x7f083c06ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49424/0 (socket says 172.21.15.90:49424) 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 -- 172.21.15.90:0/3423663571 learned_addr learned my addr 172.21.15.90:0/3423663571 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083be00640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c15c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 -- 172.21.15.90:0/3423663571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 msgr2=0x7f083c15c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c15c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 -- 172.21.15.90:0/3423663571 >> v1:172.21.15.90:6789/0 conn(0x7f083c0cb630 legacy=0x7f083c160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 -- 172.21.15.90:0/3423663571 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c15d320 con 0x7f083c15be10 2024-03-31T15:57:06.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083aa00640 1 -- 172.21.15.90:0/3423663571 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2616238884 0 0) 0x7f083c15cd60 con 0x7f083c0cb630 2024-03-31T15:57:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083b400640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c06ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f082c009870 tx=0x7f082c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ba01bb82a9213d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083aa00640 1 -- 172.21.15.90:0/3423663571 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c0b1070 con 0x7f083c15be10 2024-03-31T15:57:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083aa00640 1 -- 172.21.15.90:0/3423663571 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f082c0a8050 con 0x7f083c15be10 2024-03-31T15:57:06.799 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f083aa00640 1 -- 172.21.15.90:0/3423663571 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c0ac860 con 0x7f083c15be10 2024-03-31T15:57:06.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f0842400640 1 -- 172.21.15.90:0/3423663571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 msgr2=0x7f083c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.803+0000 7f0842400640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c06ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f082c009870 tx=0x7f082c01c4d0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.800 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- 172.21.15.90:0/3423663571 shutdown_connections 2024-03-31T15:57:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c15c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 --2- 172.21.15.90:0/3423663571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c06ea10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- 172.21.15.90:0/3423663571 >> 172.21.15.90:0/3423663571 conn(0x7f083c0b1880 msgr2=0x7f083c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- 172.21.15.90:0/3423663571 shutdown_connections 2024-03-31T15:57:06.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- 172.21.15.90:0/3423663571 wait complete. 2024-03-31T15:57:06.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 Processor -- start 2024-03-31T15:57:06.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- start start 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f083c0cb630 0x7f083c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f083c169160 con 0x7f083c15be10 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f083c1692e0 con 0x7f083c0cb630 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f083c169460 con 0x7f083c0cbae0 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49426/0 (socket says 172.21.15.90:49426) 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 -- 172.21.15.90:0/2403508978 learned_addr learned my addr 172.21.15.90:0/2403508978 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083b400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f083c0cb630 0x7f083c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0832a00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 -- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 msgr2=0x7f083c158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f083c0cbae0 0x7f083c158f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 -- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f083c0cb630 msgr2=0x7f083c14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f083c0cb630 0x7f083c14dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c15a9a0 con 0x7f083c15be10 2024-03-31T15:57:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f083be00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c154bc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0834007e70 tx=0x7f083400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834014620 con 0x7f083c15be10 2024-03-31T15:57:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834019070 con 0x7f083c15be10 2024-03-31T15:57:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.807+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083c15ac40 con 0x7f083c15be10 2024-03-31T15:57:06.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834014a20 con 0x7f083c15be10 2024-03-31T15:57:06.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083c15b200 con 0x7f083c15be10 2024-03-31T15:57:06.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f083c0686d0 con 0x7f083c15be10 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0834014bc0 con 0x7f083c15be10 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0818068380 0x7f081806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f0834087240 con 0x7f083c15be10 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7f081806e820 0x7f0818070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0834006ba0 con 0x7f081806e820 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f083408e050 con 0x7f083c15be10 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0832a00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7f081806e820 0x7f0818070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f083b400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0818068380 0x7f081806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.811+0000 7f0832a00640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7f081806e820 0x7f0818070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.815+0000 7f083b400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0818068380 0x7f081806a840 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f082c0ae000 tx=0x7f082c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.815+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== osd.2 v2:172.21.15.90:6800/347246527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0834006ba0 con 0x7f081806e820 2024-03-31T15:57:06.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 --> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f083c06e6a0 con 0x7f081806e820 2024-03-31T15:57:06.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0833e00640 1 -- 172.21.15.90:0/2403508978 <== osd.2 v2:172.21.15.90:6800/347246527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f083c06e6a0 con 0x7f081806e820 2024-03-31T15:57:06.821 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:06.821 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:06.822 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7f081806e820 msgr2=0x7f0818070c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:6800/347246527,v1:172.21.15.90:6801/347246527] conn(0x7f081806e820 0x7f0818070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0818068380 msgr2=0x7f081806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f0818068380 0x7f081806a840 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f082c0ae000 tx=0x7f082c0a7000 comp rx=0 tx=0).stop 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 msgr2=0x7f083c154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 --2- 172.21.15.90:0/2403508978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f083c15be10 0x7f083c154bc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0834007e70 tx=0x7f083400e4b0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f083be00640 1 -- 172.21.15.90:0/2403508978 reap_dead start 2024-03-31T15:57:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 shutdown_connections 2024-03-31T15:57:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 >> 172.21.15.90:0/2403508978 conn(0x7f083c0b1880 msgr2=0x7f083c0cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 shutdown_connections 2024-03-31T15:57:06.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.827+0000 7f0842400640 1 -- 172.21.15.90:0/2403508978 wait complete. 2024-03-31T15:57:06.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-31T15:57:06.975 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 Processor -- start 2024-03-31T15:57:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 -- start start 2024-03-31T15:57:06.976 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa124151d70 0x7fa1240cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa12415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa124071cd0 con 0x7fa124072a90 2024-03-31T15:57:06.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa124071fb0 con 0x7fa124151d70 2024-03-31T15:57:06.977 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa129e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa124072290 con 0x7fa124152150 2024-03-31T15:57:06.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa124151d70 0x7fa1240cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa124151d70 0x7fa1240cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49442/0 (socket says 172.21.15.90:49442) 2024-03-31T15:57:06.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 -- 172.21.15.90:0/519018852 learned_addr learned my addr 172.21.15.90:0/519018852 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.978 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa11a000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa124072a90 0x7fa12415efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50082/0 (socket says 172.21.15.90:50082) 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa123400640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa12415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 -- 172.21.15.90:0/519018852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 msgr2=0x7fa12415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa12415f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 -- 172.21.15.90:0/519018852 >> v1:172.21.15.90:6789/0 conn(0x7fa124072a90 legacy=0x7fa12415efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122a00640 1 -- 172.21.15.90:0/519018852 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa124072570 con 0x7fa124151d70 2024-03-31T15:57:06.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.983+0000 7fa122000640 1 -- 172.21.15.90:0/519018852 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487015494 0 0) 0x7fa124071cd0 con 0x7fa124072a90 2024-03-31T15:57:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa122a00640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa1240cc440 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa114009870 tx=0x7fa11401cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b919bd6fad775ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.980 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa122000640 1 -- 172.21.15.90:0/519018852 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140a9640 con 0x7fa124151d70 2024-03-31T15:57:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa122000640 1 -- 172.21.15.90:0/519018852 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1140a97e0 con 0x7fa124151d70 2024-03-31T15:57:06.981 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa122000640 1 -- 172.21.15.90:0/519018852 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140a9bb0 con 0x7fa124151d70 2024-03-31T15:57:06.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- 172.21.15.90:0/519018852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 msgr2=0x7fa1240cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa1240cc440 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa114009870 tx=0x7fa11401cfc0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- 172.21.15.90:0/519018852 shutdown_connections 2024-03-31T15:57:06.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa12415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 --2- 172.21.15.90:0/519018852 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa1240cc440 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- 172.21.15.90:0/519018852 >> 172.21.15.90:0/519018852 conn(0x7fa1240b1a50 msgr2=0x7fa1240b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:06.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- 172.21.15.90:0/519018852 shutdown_connections 2024-03-31T15:57:06.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- 172.21.15.90:0/519018852 wait complete. 2024-03-31T15:57:06.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 Processor -- start 2024-03-31T15:57:06.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.987+0000 7fa129e00640 1 -- start start 2024-03-31T15:57:06.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa124072a90 0x7fa124147760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa1241539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1240ca380 con 0x7fa124151d70 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa12415c370 con 0x7fa124072a90 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa1240ca500 con 0x7fa124152150 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49448/0 (socket says 172.21.15.90:49448) 2024-03-31T15:57:06.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 -- 172.21.15.90:0/3191448894 learned_addr learned my addr 172.21.15.90:0/3191448894 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa123400640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa1241539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa122a00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa124072a90 0x7fa124147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 -- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 msgr2=0x7fa1241539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa1241539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 -- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa124072a90 msgr2=0x7fa124147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa124072a90 0x7fa124147760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 -- 172.21.15.90:0/3191448894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1240b2ff0 con 0x7fa124151d70 2024-03-31T15:57:06.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11a000640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0011f0 tx=0x7fa10c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c018070 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10c0074e0 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1240b32d0 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c004a10 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1240b3890 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa124068660 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.991+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa10c01b070 con 0x7fa124151d70 2024-03-31T15:57:06.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.995+0000 7fa11be00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 0x7fa10006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:06.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.995+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fa10c085c40 con 0x7fa124151d70 2024-03-31T15:57:06.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.995+0000 7fa122a00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 0x7fa10006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:06.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.995+0000 7fa122a00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 0x7fa10006a840 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ae000 tx=0x7fa1140b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:06.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:06.999+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa10c0562d0 con 0x7fa124151d70 2024-03-31T15:57:07.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fa1241537b0 con 0x7fa100068380 2024-03-31T15:57:07.152 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa11be00640 1 -- 172.21.15.90:0/3191448894 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa1241537b0 con 0x7fa100068380 2024-03-31T15:57:07.153 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 2 to deep-scrub 2024-03-31T15:57:07.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 msgr2=0x7fa10006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.154 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 0x7fa10006a840 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa1140ae000 tx=0x7fa1140b3040 comp rx=0 tx=0).stop 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 msgr2=0x7fa124152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0011f0 tx=0x7fa10c00c9b0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 shutdown_connections 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa100068380 0x7fa10006a840 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa124152150 0x7fa1241539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa124151d70 0x7fa124152f70 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 --2- 172.21.15.90:0/3191448894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa124072a90 0x7fa124147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 >> 172.21.15.90:0/3191448894 conn(0x7fa1240b1a50 msgr2=0x7fa12406d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 shutdown_connections 2024-03-31T15:57:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.159+0000 7fa129e00640 1 -- 172.21.15.90:0/3191448894 wait complete. 2024-03-31T15:57:07.173 INFO:tasks.ceph:Scrubbing osd.3 2024-03-31T15:57:07.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 Processor -- start 2024-03-31T15:57:07.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 -- start start 2024-03-31T15:57:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f8664162460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f866406da60 0x7f866406def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86640b4260 con 0x7f866415e3b0 2024-03-31T15:57:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f86640ccd10 con 0x7f866406da60 2024-03-31T15:57:07.314 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8669a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f86641629a0 con 0x7f8664162000 2024-03-31T15:57:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8662a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f866415e3b0 0x7f86640cc600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50098/0 (socket says 172.21.15.90:50098) 2024-03-31T15:57:07.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8662a00640 1 -- 172.21.15.90:0/771232540 learned_addr learned my addr 172.21.15.90:0/771232540 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:07.316 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790403204 0 0) 0x7f86640b4260 con 0x7f866415e3b0 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8648003530 con 0x7f866415e3b0 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8662000640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f8664162460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8663400640 1 --2- 172.21.15.90:0/771232540 >> v2:172.21.15.102:3300/0 conn(0x7f866406da60 0x7f866406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4207588671 0 0) 0x7f8648003530 con 0x7f866415e3b0 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 msgr2=0x7f8664162460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f8664162460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 >> v2:172.21.15.102:3300/0 conn(0x7f866406da60 msgr2=0x7f866406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 --2- 172.21.15.90:0/771232540 >> v2:172.21.15.102:3300/0 conn(0x7f866406da60 0x7f866406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86640b4540 con 0x7f866415e3b0 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f8654001680 con 0x7f866415e3b0 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 >> v1:172.21.15.90:6789/0 conn(0x7f866415e3b0 legacy=0x7f86640cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f86480039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648003ff0 0x7f8648004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8648004d50 0x7f86480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f86640ccd10 con 0x7f8648004d50 2024-03-31T15:57:07.318 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f864800ba80 con 0x7f8648003ff0 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f864800bc50 con 0x7f8664162000 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8663400640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8648004d50 0x7f86480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.319+0000 7f8662000640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648003ff0 0x7f8648004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8662a00640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f86480039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 -- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 msgr2=0x7f86480039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8664162000 0x7f86480039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 -- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648003ff0 msgr2=0x7f8648004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648003ff0 0x7f8648004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 -- 172.21.15.90:0/771232540 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8648003530 con 0x7f8648004d50 2024-03-31T15:57:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8648004d50 0x7f86480051d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f865c007360 tx=0x7f865c01ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d5dddfce5c15ac server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0b2070 con 0x7f8648004d50 2024-03-31T15:57:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865c0a5d80 con 0x7f8648004d50 2024-03-31T15:57:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8661600640 1 -- 172.21.15.90:0/771232540 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c009be0 con 0x7f8648004d50 2024-03-31T15:57:07.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8669a00640 1 -- 172.21.15.90:0/771232540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8648004d50 msgr2=0x7f86480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8669a00640 1 --2- 172.21.15.90:0/771232540 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8648004d50 0x7f86480051d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f865c007360 tx=0x7f865c01ce80 comp rx=0 tx=0).stop 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8663400640 1 -- 172.21.15.90:0/771232540 reap_dead start 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8669a00640 1 -- 172.21.15.90:0/771232540 shutdown_connections 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.323+0000 7f8669a00640 1 -- 172.21.15.90:0/771232540 >> 172.21.15.90:0/771232540 conn(0x7f866409aa40 msgr2=0x7f866409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- 172.21.15.90:0/771232540 shutdown_connections 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- 172.21.15.90:0/771232540 wait complete. 2024-03-31T15:57:07.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 Processor -- start 2024-03-31T15:57:07.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- start start 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648004d50 0x7f8664156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 0x7f866415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f86640b6da0 0x7f86640b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f866406d260 con 0x7f86641602e0 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f866406d5d0 con 0x7f8648004d50 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f866406e430 con 0x7f86640b6da0 2024-03-31T15:57:07.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 0x7f866415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 0x7f866415e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49460/0 (socket says 172.21.15.90:49460) 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 -- 172.21.15.90:0/254237431 learned_addr learned my addr 172.21.15.90:0/254237431 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648004d50 0x7f8664156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8663400640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f86640b6da0 0x7f86640b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 -- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f86640b6da0 msgr2=0x7f86640b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f86640b6da0 0x7f86640b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 -- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648004d50 msgr2=0x7f8664156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648004d50 0x7f8664156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866415efe0 con 0x7f86641602e0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8648004d50 0x7f8664156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8663400640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f86640b6da0 0x7f86640b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8662000640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 0x7f866415e790 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f865000b7f0 tx=0x7f865000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650018070 con 0x7f86641602e0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86500107d0 con 0x7f86641602e0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866415f2c0 con 0x7f86641602e0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.327+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650010ba0 con 0x7f86641602e0 2024-03-31T15:57:07.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86640c92d0 con 0x7f86641602e0 2024-03-31T15:57:07.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86640686d0 con 0x7f86641602e0 2024-03-31T15:57:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f865001b070 con 0x7f86641602e0 2024-03-31T15:57:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f86440683b0 0x7f864406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f8650087330 con 0x7f86641602e0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7f864406e850 0x7f8644070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f865000ee90 con 0x7f864406e850 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f86500876e0 con 0x7f86641602e0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8662a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f86440683b0 0x7f864406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8663400640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7f864406e850 0x7f8644070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8663400640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7f864406e850 0x7f8644070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.331+0000 7f8662a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f86440683b0 0x7f864406a870 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f86640c8f80 tx=0x7f8654009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.335+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== osd.3 v2:172.21.15.90:6808/4045683431 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f865000ee90 con 0x7f864406e850 2024-03-31T15:57:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 --> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8664160a50 con 0x7f864406e850 2024-03-31T15:57:07.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f865aa00640 1 -- 172.21.15.90:0/254237431 <== osd.3 v2:172.21.15.90:6808/4045683431 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8664160a50 con 0x7f864406e850 2024-03-31T15:57:07.341 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:07.342 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:07.342 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7f864406e850 msgr2=0x7f8644070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:6808/4045683431,v1:172.21.15.90:6810/4045683431] conn(0x7f864406e850 0x7f8644070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f86440683b0 msgr2=0x7f864406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f86440683b0 0x7f864406a870 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f86640c8f80 tx=0x7f8654009040 comp rx=0 tx=0).stop 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 msgr2=0x7f866415e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 --2- 172.21.15.90:0/254237431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86641602e0 0x7f866415e790 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f865000b7f0 tx=0x7f865000bcb0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8663400640 1 -- 172.21.15.90:0/254237431 reap_dead start 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 shutdown_connections 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 >> 172.21.15.90:0/254237431 conn(0x7f866409aa40 msgr2=0x7f86640cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 shutdown_connections 2024-03-31T15:57:07.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.347+0000 7f8669a00640 1 -- 172.21.15.90:0/254237431 wait complete. 2024-03-31T15:57:07.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-31T15:57:07.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 Processor -- start 2024-03-31T15:57:07.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 -- start start 2024-03-31T15:57:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fec166fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fec0d0ea0 con 0x7f6fec0ce910 2024-03-31T15:57:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fec0cd460 con 0x7f6fec15d240 2024-03-31T15:57:07.497 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6ff2800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6fec0cd220 con 0x7f6fec164be0 2024-03-31T15:57:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6feb400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f6fec0ce910 0x7f6fec0cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50104/0 (socket says 172.21.15.90:50104) 2024-03-31T15:57:07.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6feb400640 1 -- 172.21.15.90:0/3514952724 learned_addr learned my addr 172.21.15.90:0/3514952724 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:07.499 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566190509 0 0) 0x7f6fec0d0ea0 con 0x7f6fec0ce910 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd0003530 con 0x7f6fec0ce910 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fec166fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2267953787 0 0) 0x7f6fd0003530 con 0x7f6fec0ce910 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 msgr2=0x7f6fec166fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fec166fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 >> v1:172.21.15.102:6789/0 conn(0x7f6fec15d240 legacy=0x7f6fec0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec0cd740 con 0x7f6fec0ce910 2024-03-31T15:57:07.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7f6fdc001680 con 0x7f6fec0ce910 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 >> v1:172.21.15.90:6789/0 conn(0x7f6fec0ce910 legacy=0x7f6fec0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fd00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd0004020 0x7f6fd0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fd0004d50 0x7f6fd00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec0cd220 con 0x7f6fd0004d50 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd000bc10 con 0x7f6fd0004020 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd000be10 con 0x7f6fec164be0 2024-03-31T15:57:07.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fd0004d50 0x7f6fd00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6feaa00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd0004020 0x7f6fd0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6feb400640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fd00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 -- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 msgr2=0x7f6fd00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec164be0 0x7f6fd00039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 -- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd0004020 msgr2=0x7f6fd0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fd0004020 0x7f6fd0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.503+0000 7f6febe00640 1 -- 172.21.15.90:0/3514952724 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd0003530 con 0x7f6fd0004d50 2024-03-31T15:57:07.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6febe00640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fd0004d50 0x7f6fd00051d0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6fec08c510 tx=0x7f6fe401cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1139e041187c691c server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40b2070 con 0x7f6fd0004d50 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe40a5ce0 con 0x7f6fd0004d50 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6fea000640 1 -- 172.21.15.90:0/3514952724 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4009be0 con 0x7f6fd0004d50 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 -- 172.21.15.90:0/3514952724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fd0004d50 msgr2=0x7f6fd00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 --2- 172.21.15.90:0/3514952724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fd0004d50 0x7f6fd00051d0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6fec08c510 tx=0x7f6fe401cde0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.504 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6febe00640 1 -- 172.21.15.90:0/3514952724 reap_dead start 2024-03-31T15:57:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 -- 172.21.15.90:0/3514952724 shutdown_connections 2024-03-31T15:57:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 -- 172.21.15.90:0/3514952724 >> 172.21.15.90:0/3514952724 conn(0x7f6fec08bfd0 msgr2=0x7f6fec0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 -- 172.21.15.90:0/3514952724 shutdown_connections 2024-03-31T15:57:07.505 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.507+0000 7f6ff2800640 1 -- 172.21.15.90:0/3514952724 wait complete. 2024-03-31T15:57:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 Processor -- start 2024-03-31T15:57:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- start start 2024-03-31T15:57:07.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fec14ea30 0x7f6fec151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec0c5a30 0x7f6fec152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec15d780 con 0x7f6fec0b5940 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec0cdd00 con 0x7f6fec14ea30 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6fec06c700 con 0x7f6fec0c5a30 2024-03-31T15:57:07.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49474/0 (socket says 172.21.15.90:49474) 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 -- 172.21.15.90:0/968656511 learned_addr learned my addr 172.21.15.90:0/968656511 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6feb400640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fec14ea30 0x7f6fec151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6feaa00640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec0c5a30 0x7f6fec152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 -- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec0c5a30 msgr2=0x7f6fec152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec0c5a30 0x7f6fec152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 -- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fec14ea30 msgr2=0x7f6fec151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fec14ea30 0x7f6fec151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 -- 172.21.15.90:0/968656511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec0b4b80 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6febe00640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4002240 tx=0x7f6fd0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40bb070 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe40a5ae0 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec0b4e60 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40b2040 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.511+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec0b5420 con 0x7f6fec0b5940 2024-03-31T15:57:07.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec068660 con 0x7f6fec0b5940 2024-03-31T15:57:07.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6fe40be070 con 0x7f6fec0b5940 2024-03-31T15:57:07.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6fe3400640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 0x7f6fcc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.511 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f6fe411ff30 con 0x7f6fec0b5940 2024-03-31T15:57:07.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6feb400640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 0x7f6fcc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.515+0000 7f6feb400640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 0x7f6fcc06a870 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc001060 tx=0x7f6fdc001b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.519+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6fe4016af0 con 0x7f6fec0b5940 2024-03-31T15:57:07.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.675+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f6fec0ce6f0 con 0x7f6fcc0683b0 2024-03-31T15:57:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.675+0000 7f6fe3400640 1 -- 172.21.15.90:0/968656511 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6fec0ce6f0 con 0x7f6fcc0683b0 2024-03-31T15:57:07.671 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 3 to deep-scrub 2024-03-31T15:57:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 msgr2=0x7f6fcc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 0x7f6fcc06a870 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc001060 tx=0x7f6fdc001b20 comp rx=0 tx=0).stop 2024-03-31T15:57:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 msgr2=0x7f6fec0b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4002240 tx=0x7f6fd0005840 comp rx=0 tx=0).stop 2024-03-31T15:57:07.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 shutdown_connections 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fcc0683b0 0x7f6fcc06a870 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6fec0b5940 0x7f6fec0b5da0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6fec0c5a30 0x7f6fec152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 --2- 172.21.15.90:0/968656511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fec14ea30 0x7f6fec151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 >> 172.21.15.90:0/968656511 conn(0x7f6fec08bfd0 msgr2=0x7f6fec06df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 shutdown_connections 2024-03-31T15:57:07.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.679+0000 7f6ff2800640 1 -- 172.21.15.90:0/968656511 wait complete. 2024-03-31T15:57:07.692 INFO:tasks.ceph:Scrubbing osd.4 2024-03-31T15:57:07.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:07.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.835+0000 7fcf3b600640 1 Processor -- start 2024-03-31T15:57:07.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.835+0000 7fcf3b600640 1 -- start start 2024-03-31T15:57:07.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3415b150 0x7fcf3406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415eba0 0x7fcf340ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcf340caf20 0x7fcf3415bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3415b850 con 0x7fcf3415b150 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3415b530 con 0x7fcf340caf20 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf3b600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3415f310 con 0x7fcf3415eba0 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3415b150 0x7fcf3406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fcf3415b150 0x7fcf3406eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49478/0 (socket says 172.21.15.90:49478) 2024-03-31T15:57:07.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 -- 172.21.15.90:0/3405079462 learned_addr learned my addr 172.21.15.90:0/3405079462 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf39600640 1 --2- 172.21.15.90:0/3405079462 >> v2:172.21.15.102:3300/0 conn(0x7fcf340caf20 0x7fcf3415bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf33e00640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415eba0 0x7fcf340ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 -- 172.21.15.90:0/3405079462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415eba0 msgr2=0x7fcf340ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415eba0 0x7fcf340ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 -- 172.21.15.90:0/3405079462 >> v2:172.21.15.102:3300/0 conn(0x7fcf340caf20 msgr2=0x7fcf3415bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 --2- 172.21.15.90:0/3405079462 >> v2:172.21.15.102:3300/0 conn(0x7fcf340caf20 0x7fcf3415bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 -- 172.21.15.90:0/3405079462 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3415f5f0 con 0x7fcf3415b150 2024-03-31T15:57:07.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.839+0000 7fcf38c00640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3415b150 0x7fcf3406eac0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcf340cf860 tx=0x7fcf2801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c439e449d890f383 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf33400640 1 -- 172.21.15.90:0/3405079462 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b0070 con 0x7fcf3415b150 2024-03-31T15:57:07.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf33400640 1 -- 172.21.15.90:0/3405079462 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf280ab600 con 0x7fcf3415b150 2024-03-31T15:57:07.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf33400640 1 -- 172.21.15.90:0/3405079462 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280ab9d0 con 0x7fcf3415b150 2024-03-31T15:57:07.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- 172.21.15.90:0/3405079462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3415b150 msgr2=0x7fcf3406eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3415b150 0x7fcf3406eac0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcf340cf860 tx=0x7fcf2801c520 comp rx=0 tx=0).stop 2024-03-31T15:57:07.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- 172.21.15.90:0/3405079462 shutdown_connections 2024-03-31T15:57:07.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- 172.21.15.90:0/3405079462 >> v2:172.21.15.102:3300/0 conn(0x7fcf340caf20 0x7fcf3415bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415eba0 0x7fcf340ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- 172.21.15.90:0/3405079462 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf3415b150 0x7fcf3406eac0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- 172.21.15.90:0/3405079462 >> 172.21.15.90:0/3405079462 conn(0x7fcf340b1880 msgr2=0x7fcf340b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- 172.21.15.90:0/3405079462 shutdown_connections 2024-03-31T15:57:07.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- 172.21.15.90:0/3405079462 wait complete. 2024-03-31T15:57:07.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 Processor -- start 2024-03-31T15:57:07.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- start start 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 0x7fcf341537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415b150 0x7fcf34153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf3415eba0 0x7fcf340b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcf340ce4e0 con 0x7fcf340caf20 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcf340ce360 con 0x7fcf3415eba0 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf3b600640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fcf340ce660 con 0x7fcf3415b150 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf38c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 0x7fcf341537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf38c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 0x7fcf341537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49492/0 (socket says 172.21.15.90:49492) 2024-03-31T15:57:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.843+0000 7fcf38c00640 1 -- 172.21.15.90:0/1160400805 learned_addr learned my addr 172.21.15.90:0/1160400805 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf33e00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415b150 0x7fcf34153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf39600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf3415eba0 0x7fcf340b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 -- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415b150 msgr2=0x7fcf34153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.843 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fcf3415b150 0x7fcf34153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 -- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf3415eba0 msgr2=0x7fcf340b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf3415eba0 0x7fcf340b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf34154360 con 0x7fcf340caf20 2024-03-31T15:57:07.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf38c00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 0x7fcf341537c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ae040 tx=0x7fcf2801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b0040 con 0x7fcf340caf20 2024-03-31T15:57:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf280bc070 con 0x7fcf340caf20 2024-03-31T15:57:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf34154640 con 0x7fcf340caf20 2024-03-31T15:57:07.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b6680 con 0x7fcf340caf20 2024-03-31T15:57:07.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf34154c00 con 0x7fcf340caf20 2024-03-31T15:57:07.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcf340686d0 con 0x7fcf340caf20 2024-03-31T15:57:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcf280b6820 con 0x7fcf340caf20 2024-03-31T15:57:07.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcf140683b0 0x7fcf1406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fcf28121580 con 0x7fcf340caf20 2024-03-31T15:57:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7fcf1406e850 0x7fcf14070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf280ab600 con 0x7fcf1406e850 2024-03-31T15:57:07.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.847+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fcf28121a10 con 0x7fcf340caf20 2024-03-31T15:57:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.851+0000 7fcf33e00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcf140683b0 0x7fcf1406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.851+0000 7fcf39600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7fcf1406e850 0x7fcf14070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.851+0000 7fcf33e00640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcf140683b0 0x7fcf1406a870 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcf34165d90 tx=0x7fcf20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.851+0000 7fcf39600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7fcf1406e850 0x7fcf14070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:07.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.851+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== osd.4 v2:172.21.15.102:6800/101321808 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf280ab600 con 0x7fcf1406e850 2024-03-31T15:57:07.858 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 --> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcf340cdf30 con 0x7fcf1406e850 2024-03-31T15:57:07.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf30c00640 1 -- 172.21.15.90:0/1160400805 <== osd.4 v2:172.21.15.102:6800/101321808 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcf340cdf30 con 0x7fcf1406e850 2024-03-31T15:57:07.859 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:07.859 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:07.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7fcf1406e850 msgr2=0x7fcf14070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6800/101321808,v1:172.21.15.102:6801/101321808] conn(0x7fcf1406e850 0x7fcf14070c90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:07.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcf140683b0 msgr2=0x7fcf1406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fcf140683b0 0x7fcf1406a870 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fcf34165d90 tx=0x7fcf20008040 comp rx=0 tx=0).stop 2024-03-31T15:57:07.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 msgr2=0x7fcf341537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:07.861 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf3b600640 1 --2- 172.21.15.90:0/1160400805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcf340caf20 0x7fcf341537c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ae040 tx=0x7fcf2801ad60 comp rx=0 tx=0).stop 2024-03-31T15:57:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.863+0000 7fcf39600640 1 -- 172.21.15.90:0/1160400805 reap_dead start 2024-03-31T15:57:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.867+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 shutdown_connections 2024-03-31T15:57:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.867+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 >> 172.21.15.90:0/1160400805 conn(0x7fcf340b1880 msgr2=0x7fcf3414f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.867+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 shutdown_connections 2024-03-31T15:57:07.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:07.867+0000 7fcf3b600640 1 -- 172.21.15.90:0/1160400805 wait complete. 2024-03-31T15:57:07.877 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-31T15:57:08.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 Processor -- start 2024-03-31T15:57:08.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 -- start start 2024-03-31T15:57:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 0x7f6bac0cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bac0d0ea0 con 0x7f6bac15d240 2024-03-31T15:57:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6bac0cd220 con 0x7f6bac0ce910 2024-03-31T15:57:08.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6bb0e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6bac0cf230 con 0x7f6bac163bd0 2024-03-31T15:57:08.015 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3301/0 says I am v2:172.21.15.90:40296/0 (socket says 172.21.15.90:40296) 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6ba9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6bac15d240 0x7f6bac0ccb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42484/0 (socket says 172.21.15.90:42484) 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 -- 172.21.15.90:0/3956743845 learned_addr learned my addr 172.21.15.90:0/3956743845 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6ba9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 0x7f6bac0cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6ba8a00640 1 -- 172.21.15.90:0/3956743845 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645989577 0 0) 0x7f6bac0d0ea0 con 0x7f6bac15d240 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6ba8a00640 1 -- 172.21.15.90:0/3956743845 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b90003530 con 0x7f6bac15d240 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 -- 172.21.15.90:0/3956743845 >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 msgr2=0x7f6bac0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.016 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 --2- 172.21.15.90:0/3956743845 >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 0x7f6bac0cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 -- 172.21.15.90:0/3956743845 >> v1:172.21.15.102:6789/0 conn(0x7f6bac15d240 legacy=0x7f6bac0ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6baa800640 1 -- 172.21.15.90:0/3956743845 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac0cd460 con 0x7f6bac163bd0 2024-03-31T15:57:08.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.019+0000 7f6ba9e00640 1 --2- 172.21.15.90:0/3956743845 >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 0x7f6bac0cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:57:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6baa800640 1 --2- 172.21.15.90:0/3956743845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f35d1648ff3e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6ba8a00640 1 -- 172.21.15.90:0/3956743845 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40a5850 con 0x7f6bac163bd0 2024-03-31T15:57:08.019 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6ba8a00640 1 -- 172.21.15.90:0/3956743845 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ba40a59f0 con 0x7f6bac163bd0 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6ba8a00640 1 -- 172.21.15.90:0/3956743845 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40a5dc0 con 0x7f6bac163bd0 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 -- 172.21.15.90:0/3956743845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 msgr2=0x7f6bac165fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/3956743845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba401c970 comp rx=0 tx=0).stop 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 -- 172.21.15.90:0/3956743845 shutdown_connections 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/3956743845 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac165fc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/3956743845 >> v2:172.21.15.102:3300/0 conn(0x7f6bac0ce910 0x7f6bac0cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 -- 172.21.15.90:0/3956743845 >> 172.21.15.90:0/3956743845 conn(0x7f6bac08bfd0 msgr2=0x7f6bac0d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 -- 172.21.15.90:0/3956743845 shutdown_connections 2024-03-31T15:57:08.020 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.023+0000 7f6bb0e00640 1 -- 172.21.15.90:0/3956743845 wait complete. 2024-03-31T15:57:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 Processor -- start 2024-03-31T15:57:08.022 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- start start 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bac0ce910 0x7f6bac15cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac1631e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac168d20 con 0x7f6bac15d240 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac15d780 con 0x7f6bac0ce910 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6bac168ea0 con 0x7f6bac163bd0 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49494/0 (socket says 172.21.15.90:49494) 2024-03-31T15:57:08.023 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 -- 172.21.15.90:0/1984228714 learned_addr learned my addr 172.21.15.90:0/1984228714 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bac0ce910 0x7f6bac15cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6baa800640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac1631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 -- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 msgr2=0x7f6bac1631e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac1631e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 -- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bac0ce910 msgr2=0x7f6bac15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bac0ce910 0x7f6bac15cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 -- 172.21.15.90:0/1984228714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac0b5060 con 0x7f6bac15d240 2024-03-31T15:57:08.024 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba9400640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00c460 tx=0x7f6b9c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c018070 con 0x7f6bac15d240 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b9c0074a0 con 0x7f6bac15d240 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c004a30 con 0x7f6bac15d240 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bac15be60 con 0x7f6bac15d240 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.027+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bac158c70 con 0x7f6bac15d240 2024-03-31T15:57:08.026 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bac068660 con 0x7f6bac15d240 2024-03-31T15:57:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b9c004bd0 con 0x7f6bac15d240 2024-03-31T15:57:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6ba2000640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 0x7f6b8806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f6b9c085a60 con 0x7f6bac15d240 2024-03-31T15:57:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6ba9e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 0x7f6b8806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.031+0000 7f6ba9e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 0x7f6b8806a840 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009580 tx=0x7f6b94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.035+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f6b9c055b40 con 0x7f6bac15d240 2024-03-31T15:57:08.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f6bac0ce6f0 con 0x7f6b88068380 2024-03-31T15:57:08.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6ba2000640 1 -- 172.21.15.90:0/1984228714 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6bac0ce6f0 con 0x7f6b88068380 2024-03-31T15:57:08.189 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 4 to deep-scrub 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 msgr2=0x7f6b8806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 0x7f6b8806a840 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009580 tx=0x7f6b94009190 comp rx=0 tx=0).stop 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 msgr2=0x7f6bac0b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00c460 tx=0x7f6b9c00c920 comp rx=0 tx=0).stop 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 shutdown_connections 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6b88068380 0x7f6b8806a840 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f6bac163bd0 0x7f6bac1631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac15d240 0x7f6bac0b4b20 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 --2- 172.21.15.90:0/1984228714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bac0ce910 0x7f6bac15cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 >> 172.21.15.90:0/1984228714 conn(0x7f6bac08bfd0 msgr2=0x7f6bac156820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 shutdown_connections 2024-03-31T15:57:08.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.195+0000 7f6bb0e00640 1 -- 172.21.15.90:0/1984228714 wait complete. 2024-03-31T15:57:08.210 INFO:tasks.ceph:Scrubbing osd.5 2024-03-31T15:57:08.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:08.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 Processor -- start 2024-03-31T15:57:08.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 -- start start 2024-03-31T15:57:08.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7fabc00cc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc015b530 con 0x7fabc006eac0 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc015f310 con 0x7fabc015b150 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabc5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fabc015b850 con 0x7fabc015eba0 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabb6000640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fabc006eac0 0x7fabc015bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50114/0 (socket says 172.21.15.90:50114) 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabb6000640 1 -- 172.21.15.90:0/3011939003 learned_addr learned my addr 172.21.15.90:0/3011939003 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7fabc00cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 761867237 0 0) 0x7fabc015b530 con 0x7fabc006eac0 2024-03-31T15:57:08.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faba4005740 con 0x7fabc006eac0 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765098391 0 0) 0x7fabc015f310 con 0x7fabc015b150 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabc015b530 con 0x7fabc015b150 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2977613730 0 0) 0x7faba4005740 con 0x7fabc006eac0 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 msgr2=0x7fabc00cc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7fabc00cc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 >> v1:172.21.15.102:6789/0 conn(0x7fabc015b150 legacy=0x7fabc00ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc015f5f0 con 0x7fabc006eac0 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7faba8001680 con 0x7fabc006eac0 2024-03-31T15:57:08.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7fabc00cc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 >> v1:172.21.15.90:6789/0 conn(0x7fabc006eac0 legacy=0x7fabc015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba4005d90 0x7faba4006190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7faba4006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4006f60 0x7faba40073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabc015b850 con 0x7faba4006f60 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faba400dc90 con 0x7faba4005d90 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7faba400de90 con 0x7fabc015eba0 2024-03-31T15:57:08.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4006f60 0x7faba40073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabb6000640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7faba4006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbea00640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba4005d90 0x7faba4006190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 -- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 msgr2=0x7faba4006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fabc015eba0 0x7faba4006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 -- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba4005d90 msgr2=0x7faba4006190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba4005d90 0x7faba4006190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.359+0000 7fabbf400640 1 -- 172.21.15.90:0/3011939003 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba4005740 con 0x7faba4006f60 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabbf400640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4006f60 0x7faba40073e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e350 tx=0x7fabb801cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892431b729e95c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b1070 con 0x7faba4006f60 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb80ac600 con 0x7faba4006f60 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabbe000640 1 -- 172.21.15.90:0/3011939003 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8009de0 con 0x7faba4006f60 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 -- 172.21.15.90:0/3011939003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4006f60 msgr2=0x7faba40073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 --2- 172.21.15.90:0/3011939003 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faba4006f60 0x7faba40073e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e350 tx=0x7fabb801cae0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabbf400640 1 -- 172.21.15.90:0/3011939003 reap_dead start 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 -- 172.21.15.90:0/3011939003 shutdown_connections 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 -- 172.21.15.90:0/3011939003 >> 172.21.15.90:0/3011939003 conn(0x7fabc00b1880 msgr2=0x7fabc00b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 -- 172.21.15.90:0/3011939003 shutdown_connections 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.363+0000 7fabc5c00640 1 -- 172.21.15.90:0/3011939003 wait complete. 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 Processor -- start 2024-03-31T15:57:08.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- start start 2024-03-31T15:57:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faba4006f60 0x7fabc015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 0x7fabc00c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc006d830 0x7fabc006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fabc015f010 con 0x7fabc00c88f0 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fabc006ef50 con 0x7fabc006d830 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fabc00cb0f0 con 0x7faba4006f60 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 0x7fabc00c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 0x7fabc00c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49510/0 (socket says 172.21.15.90:49510) 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 -- 172.21.15.90:0/275446276 learned_addr learned my addr 172.21.15.90:0/275446276 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabbea00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faba4006f60 0x7fabc015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabbf400640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc006d830 0x7fabc006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 -- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faba4006f60 msgr2=0x7fabc015a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7faba4006f60 0x7fabc015a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 -- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc006d830 msgr2=0x7fabc006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc006d830 0x7fabc006dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc0158500 con 0x7fabc00c88f0 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb6000640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 0x7fabc00c8d50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7faba80119a0 tx=0x7faba8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8010040 con 0x7fabc00c88f0 2024-03-31T15:57:08.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba801a070 con 0x7fabc00c88f0 2024-03-31T15:57:08.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc01587e0 con 0x7fabc00c88f0 2024-03-31T15:57:08.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8015410 con 0x7fabc00c88f0 2024-03-31T15:57:08.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.367+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc0158da0 con 0x7fabc00c88f0 2024-03-31T15:57:08.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fabc00686d0 con 0x7fabc00c88f0 2024-03-31T15:57:08.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faba80155b0 con 0x7fabc00c88f0 2024-03-31T15:57:08.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab9c068380 0x7fab9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7faba808a480 con 0x7fabc00c88f0 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7fab9c06e820 0x7fab9c070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab9c071330 con 0x7fab9c06e820 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7faba808a830 con 0x7fabc00c88f0 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabbea00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab9c068380 0x7fab9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabbf400640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7fab9c06e820 0x7fab9c070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabbea00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab9c068380 0x7fab9c06a840 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fabb0007af0 tx=0x7fabb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.369 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.371+0000 7fabbf400640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7fab9c06e820 0x7fab9c070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.375+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== osd.5 v2:172.21.15.102:6808/3785791459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab9c071330 con 0x7fab9c06e820 2024-03-31T15:57:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 --> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fabc00cefa0 con 0x7fab9c06e820 2024-03-31T15:57:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabb7400640 1 -- 172.21.15.90:0/275446276 <== osd.5 v2:172.21.15.102:6808/3785791459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fabc00cefa0 con 0x7fab9c06e820 2024-03-31T15:57:08.382 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:08.382 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:08.382 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7fab9c06e820 msgr2=0x7fab9c070c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6808/3785791459,v1:172.21.15.102:6810/3785791459] conn(0x7fab9c06e820 0x7fab9c070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.383 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab9c068380 msgr2=0x7fab9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fab9c068380 0x7fab9c06a840 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fabb0007af0 tx=0x7fabb0009040 comp rx=0 tx=0).stop 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 msgr2=0x7fabc00c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 --2- 172.21.15.90:0/275446276 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fabc00c88f0 0x7fabc00c8d50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7faba80119a0 tx=0x7faba8011e60 comp rx=0 tx=0).stop 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabbf400640 1 -- 172.21.15.90:0/275446276 reap_dead start 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 shutdown_connections 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 >> 172.21.15.90:0/275446276 conn(0x7fabc00b1880 msgr2=0x7fabc0071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 shutdown_connections 2024-03-31T15:57:08.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.387+0000 7fabc5c00640 1 -- 172.21.15.90:0/275446276 wait complete. 2024-03-31T15:57:08.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-31T15:57:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 Processor -- start 2024-03-31T15:57:08.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 -- start start 2024-03-31T15:57:08.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 0x7f8f40072400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f400aff50 0x7f8f401626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f40162be0 0x7f8f4006e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f8f400b0740 con 0x7f8f400aff50 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8f4006f9e0 con 0x7f8f400cce50 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f44a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4006fcc0 con 0x7f8f40162be0 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f400aff50 0x7f8f401626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f8f400aff50 0x7f8f401626a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49514/0 (socket says 172.21.15.90:49514) 2024-03-31T15:57:08.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 -- 172.21.15.90:0/3614146555 learned_addr learned my addr 172.21.15.90:0/3614146555 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f3de00640 1 --2- 172.21.15.90:0/3614146555 >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 0x7f8f40072400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f3e800640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f40162be0 0x7f8f4006e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 -- 172.21.15.90:0/3614146555 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f40162be0 msgr2=0x7f8f4006e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f40162be0 0x7f8f4006e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 -- 172.21.15.90:0/3614146555 >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 msgr2=0x7f8f40072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 --2- 172.21.15.90:0/3614146555 >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 0x7f8f40072400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f35400640 1 -- 172.21.15.90:0/3614146555 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4006ffa0 con 0x7f8f400aff50 2024-03-31T15:57:08.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.543+0000 7f8f3de00640 1 --2- 172.21.15.90:0/3614146555 >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 0x7f8f40072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:08.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f35400640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f400aff50 0x7f8f401626a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8f4006c6d0 tx=0x7f8f2801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8774e77dfa7e7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f3d400640 1 -- 172.21.15.90:0/3614146555 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f280ab5f0 con 0x7f8f400aff50 2024-03-31T15:57:08.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f3d400640 1 -- 172.21.15.90:0/3614146555 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f280b0070 con 0x7f8f400aff50 2024-03-31T15:57:08.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f3d400640 1 -- 172.21.15.90:0/3614146555 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f280ab9b0 con 0x7f8f400aff50 2024-03-31T15:57:08.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 -- 172.21.15.90:0/3614146555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f400aff50 msgr2=0x7f8f401626a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f400aff50 0x7f8f401626a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8f4006c6d0 tx=0x7f8f2801c520 comp rx=0 tx=0).stop 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 -- 172.21.15.90:0/3614146555 shutdown_connections 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f40162be0 0x7f8f4006e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3614146555 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f400aff50 0x7f8f401626a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3614146555 >> v2:172.21.15.102:3300/0 conn(0x7f8f400cce50 0x7f8f40072400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 -- 172.21.15.90:0/3614146555 >> 172.21.15.90:0/3614146555 conn(0x7f8f4006d140 msgr2=0x7f8f400cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 -- 172.21.15.90:0/3614146555 shutdown_connections 2024-03-31T15:57:08.543 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.547+0000 7f8f44a00640 1 -- 172.21.15.90:0/3614146555 wait complete. 2024-03-31T15:57:08.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 Processor -- start 2024-03-31T15:57:08.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- start start 2024-03-31T15:57:08.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f400aff50 0x7f8f400b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f400cce50 0x7f8f40153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8f400d0be0 con 0x7f8f40162be0 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8f400d0a60 con 0x7f8f400aff50 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4015db90 con 0x7f8f400cce50 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.546 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49524/0 (socket says 172.21.15.90:49524) 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 -- 172.21.15.90:0/3901472159 learned_addr learned my addr 172.21.15.90:0/3901472159 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f35400640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f400cce50 0x7f8f40153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3de00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f400aff50 0x7f8f400b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 -- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f400cce50 msgr2=0x7f8f40153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f400cce50 0x7f8f40153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 -- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f400aff50 msgr2=0x7f8f400b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f400aff50 0x7f8f400b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.547 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 -- 172.21.15.90:0/3901472159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f40159500 con 0x7f8f40162be0 2024-03-31T15:57:08.548 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f3e800640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800d550 tx=0x7f8f3800da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380125f0 con 0x7f8f40162be0 2024-03-31T15:57:08.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f38019070 con 0x7f8f40162be0 2024-03-31T15:57:08.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f40158460 con 0x7f8f40162be0 2024-03-31T15:57:08.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f380129d0 con 0x7f8f40162be0 2024-03-31T15:57:08.549 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f40158a20 con 0x7f8f40162be0 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.551+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f40068660 con 0x7f8f40162be0 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.555+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8f38012b70 con 0x7f8f40162be0 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.555+0000 7f8f36a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 0x7f8f1c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.555+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f8f38086c20 con 0x7f8f40162be0 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.555+0000 7f8f3de00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 0x7f8f1c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.555+0000 7f8f3de00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 0x7f8f1c06a840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8f40159950 tx=0x7f8f300039a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.559+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f8f380572b0 con 0x7f8f40162be0 2024-03-31T15:57:08.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.715+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8f400b6e90 con 0x7f8f1c068380 2024-03-31T15:57:08.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.715+0000 7f8f36a00640 1 -- 172.21.15.90:0/3901472159 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8f400b6e90 con 0x7f8f1c068380 2024-03-31T15:57:08.711 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 5 to deep-scrub 2024-03-31T15:57:08.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 msgr2=0x7f8f1c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 0x7f8f1c06a840 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8f40159950 tx=0x7f8f300039a0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 msgr2=0x7f8f40154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800d550 tx=0x7f8f3800da10 comp rx=0 tx=0).stop 2024-03-31T15:57:08.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 shutdown_connections 2024-03-31T15:57:08.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f8f1c068380 0x7f8f1c06a840 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8f40162be0 0x7f8f40154460 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f8f400cce50 0x7f8f40153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 --2- 172.21.15.90:0/3901472159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f400aff50 0x7f8f400b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 >> 172.21.15.90:0/3901472159 conn(0x7f8f4006d140 msgr2=0x7f8f400b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 shutdown_connections 2024-03-31T15:57:08.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.719+0000 7f8f44a00640 1 -- 172.21.15.90:0/3901472159 wait complete. 2024-03-31T15:57:08.731 INFO:tasks.ceph:Scrubbing osd.6 2024-03-31T15:57:08.731 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.875+0000 7f7016e00640 1 Processor -- start 2024-03-31T15:57:08.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.875+0000 7f7016e00640 1 -- start start 2024-03-31T15:57:08.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f701015be10 0x7f701006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70100cb630 0x7f70100cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7010160250 0x7f70101606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f701015fdc0 con 0x7f7010160250 2024-03-31T15:57:08.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f701015cd60 con 0x7f70100cb630 2024-03-31T15:57:08.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7016e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f701015d040 con 0x7f701015be10 2024-03-31T15:57:08.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7010160250 0x7f70101606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f7010160250 0x7f70101606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49528/0 (socket says 172.21.15.90:49528) 2024-03-31T15:57:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 -- 172.21.15.90:0/3013035231 learned_addr learned my addr 172.21.15.90:0/3013035231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f700fe00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f701015be10 0x7f701006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7007400640 1 --2- 172.21.15.90:0/3013035231 >> v2:172.21.15.102:3300/0 conn(0x7f70100cb630 0x7f70100cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 -- 172.21.15.90:0/3013035231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f701015be10 msgr2=0x7f701006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f701015be10 0x7f701006ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 -- 172.21.15.90:0/3013035231 >> v2:172.21.15.102:3300/0 conn(0x7f70100cb630 msgr2=0x7f70100cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:08.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 --2- 172.21.15.90:0/3013035231 >> v2:172.21.15.102:3300/0 conn(0x7f70100cb630 0x7f70100cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 -- 172.21.15.90:0/3013035231 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701015d320 con 0x7f7010160250 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.879+0000 7f7014e00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7010160250 0x7f70101606a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f70100b2940 tx=0x7f700801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af563e05d0adc0c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f700f400640 1 -- 172.21.15.90:0/3013035231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70080a5540 con 0x7f7010160250 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f700f400640 1 -- 172.21.15.90:0/3013035231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70080a56e0 con 0x7f7010160250 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f700f400640 1 -- 172.21.15.90:0/3013035231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70080a5ab0 con 0x7f7010160250 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- 172.21.15.90:0/3013035231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7010160250 msgr2=0x7f70101606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7010160250 0x7f70101606a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f70100b2940 tx=0x7f700801c6a0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- 172.21.15.90:0/3013035231 shutdown_connections 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7010160250 0x7f70101606a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- 172.21.15.90:0/3013035231 >> v2:172.21.15.102:3300/0 conn(0x7f70100cb630 0x7f70100cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- 172.21.15.90:0/3013035231 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f701015be10 0x7f701006ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- 172.21.15.90:0/3013035231 >> 172.21.15.90:0/3013035231 conn(0x7f70100b1880 msgr2=0x7f70100b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- 172.21.15.90:0/3013035231 shutdown_connections 2024-03-31T15:57:08.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- 172.21.15.90:0/3013035231 wait complete. 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 Processor -- start 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- start start 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70100cb630 0x7f701015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 0x7f7010153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7010160250 0x7f7010158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f70100ca500 con 0x7f701015be10 2024-03-31T15:57:08.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70100ca380 con 0x7f70100cb630 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7016e00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f70100ca680 con 0x7f7010160250 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7007400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 0x7f7010153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7007400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 0x7f7010153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49536/0 (socket says 172.21.15.90:49536) 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7007400640 1 -- 172.21.15.90:0/2037292287 learned_addr learned my addr 172.21.15.90:0/2037292287 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f700fe00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70100cb630 0x7f701015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.883+0000 7f7014e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7010160250 0x7f7010158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 -- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7010160250 msgr2=0x7f7010158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.888 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f7010160250 0x7f7010158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 -- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70100cb630 msgr2=0x7f701015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70100cb630 0x7f701015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70100b2980 con 0x7f701015be10 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7007400640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 0x7f7010153af0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d800 tx=0x7f6ff800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff800b6c0 con 0x7f701015be10 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff8019070 con 0x7f701015be10 2024-03-31T15:57:08.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff800ba60 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70100b2c60 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70101592b0 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7010154030 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6ff800bc00 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fec068380 0x7f6fec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f6ff8086be0 con 0x7f701015be10 2024-03-31T15:57:08.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f6fec06e820 0x7f6fec070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fec071330 con 0x7f6fec06e820 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f6ff808b220 con 0x7f701015be10 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.887+0000 7f700fe00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fec068380 0x7f6fec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.891+0000 7f7014e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f6fec06e820 0x7f6fec070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.891+0000 7f700fe00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fec068380 0x7f6fec06a840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7000009ae0 tx=0x7f7000009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.891+0000 7f7014e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f6fec06e820 0x7f6fec070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:08.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.891+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== osd.6 v2:172.21.15.102:6824/948959501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6fec071330 con 0x7f6fec06e820 2024-03-31T15:57:08.898 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.903+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 --> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f70100686d0 con 0x7f6fec06e820 2024-03-31T15:57:08.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.903+0000 7f700cc00640 1 -- 172.21.15.90:0/2037292287 <== osd.6 v2:172.21.15.102:6824/948959501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f70100686d0 con 0x7f6fec06e820 2024-03-31T15:57:08.899 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:08.899 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:08.899 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:08.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.903+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f6fec06e820 msgr2=0x7f6fec070c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.903+0000 7f7016e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6824/948959501,v1:172.21.15.102:6825/948959501] conn(0x7f6fec06e820 0x7f6fec070c60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fec068380 msgr2=0x7f6fec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f6fec068380 0x7f6fec06a840 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7000009ae0 tx=0x7f7000009190 comp rx=0 tx=0).stop 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 msgr2=0x7f7010153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 --2- 172.21.15.90:0/2037292287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f701015be10 0x7f7010153af0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d800 tx=0x7f6ff800dcc0 comp rx=0 tx=0).stop 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7014e00640 1 -- 172.21.15.90:0/2037292287 reap_dead start 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 shutdown_connections 2024-03-31T15:57:08.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 >> 172.21.15.90:0/2037292287 conn(0x7f70100b1880 msgr2=0x7f70100b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:08.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 shutdown_connections 2024-03-31T15:57:08.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:08.907+0000 7f7016e00640 1 -- 172.21.15.90:0/2037292287 wait complete. 2024-03-31T15:57:08.917 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-31T15:57:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 Processor -- start 2024-03-31T15:57:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 -- start start 2024-03-31T15:57:09.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67000ccd80 0x7f670006e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f67000b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67000cd1e0 con 0x7f670015c1d0 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f670006edc0 con 0x7f67000ccd80 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f6706800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f67000cbd20 con 0x7f670006e950 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67000ccd80 0x7f670006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f67000ccd80 0x7f670006e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49546/0 (socket says 172.21.15.90:49546) 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 -- 172.21.15.90:0/3253352036 learned_addr learned my addr 172.21.15.90:0/3253352036 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66ff400640 1 --1- 172.21.15.90:0/3253352036 >> v1:172.21.15.102:6789/0 conn(0x7f670015c1d0 0x7f6700072d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.90:42510/0 (socket says 172.21.15.90:42510) 2024-03-31T15:57:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66ffe00640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f67000b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 -- 172.21.15.90:0/3253352036 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 msgr2=0x7f67000b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f67000b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 -- 172.21.15.90:0/3253352036 >> v1:172.21.15.102:6789/0 conn(0x7f670015c1d0 legacy=0x7f6700072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.059+0000 7f66f6a00640 1 -- 172.21.15.90:0/3253352036 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67000cbfc0 con 0x7f67000ccd80 2024-03-31T15:57:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f66f6a00640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f670006e410 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009870 tx=0x7f66e801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938a0e684b4a0697 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f66fea00640 1 -- 172.21.15.90:0/3253352036 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80b1070 con 0x7f67000ccd80 2024-03-31T15:57:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f66fea00640 1 -- 172.21.15.90:0/3253352036 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e80ac620 con 0x7f67000ccd80 2024-03-31T15:57:09.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f66fea00640 1 -- 172.21.15.90:0/3253352036 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80ac9f0 con 0x7f67000ccd80 2024-03-31T15:57:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 -- 172.21.15.90:0/3253352036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 msgr2=0x7f670006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f670006e410 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009870 tx=0x7f66e801c4d0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 -- 172.21.15.90:0/3253352036 shutdown_connections 2024-03-31T15:57:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f67000b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 --2- 172.21.15.90:0/3253352036 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f670006e410 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 -- 172.21.15.90:0/3253352036 >> 172.21.15.90:0/3253352036 conn(0x7f67000b1a50 msgr2=0x7f67000b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 -- 172.21.15.90:0/3253352036 shutdown_connections 2024-03-31T15:57:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 -- 172.21.15.90:0/3253352036 wait complete. 2024-03-31T15:57:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.063+0000 7f6706800640 1 Processor -- start 2024-03-31T15:57:09.059 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- start start 2024-03-31T15:57:09.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f6700153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.060 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f670015c1d0 0x7f670015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6700073440 con 0x7f67000ccd80 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6700071450 con 0x7f670015c1d0 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f6700168cc0 con 0x7f670006e950 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49562/0 (socket says 172.21.15.90:49562) 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 -- 172.21.15.90:0/1478620870 learned_addr learned my addr 172.21.15.90:0/1478620870 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.061 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66ff400640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f6700153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66ffe00640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f670015c1d0 0x7f670015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 -- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 msgr2=0x7f6700153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f6700153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 -- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f670015c1d0 msgr2=0x7f670015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f670015c1d0 0x7f670015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 -- 172.21.15.90:0/1478620870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6700153ac0 con 0x7f67000ccd80 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f6a00640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f66e80af040 tx=0x7f66e8019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80bc070 con 0x7f67000ccd80 2024-03-31T15:57:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e80b7430 con 0x7f67000ccd80 2024-03-31T15:57:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6700158e10 con 0x7f67000ccd80 2024-03-31T15:57:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80b1040 con 0x7f67000ccd80 2024-03-31T15:57:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67001593d0 con 0x7f67000ccd80 2024-03-31T15:57:09.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.067+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6700068660 con 0x7f67000ccd80 2024-03-31T15:57:09.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.071+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f66e80bf070 con 0x7f67000ccd80 2024-03-31T15:57:09.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.071+0000 7f66f7e00640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 0x7f66dc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.071+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f66e811ff40 con 0x7f67000ccd80 2024-03-31T15:57:09.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.071+0000 7f66ff400640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 0x7f66dc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.071+0000 7f66ff400640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 0x7f66dc06a870 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6700152620 tx=0x7f66f0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.075+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f66e8016150 con 0x7f67000ccd80 2024-03-31T15:57:09.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.231+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f670006d280 con 0x7f66dc0683b0 2024-03-31T15:57:09.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.231+0000 7f66f7e00640 1 -- 172.21.15.90:0/1478620870 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f670006d280 con 0x7f66dc0683b0 2024-03-31T15:57:09.227 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 6 to deep-scrub 2024-03-31T15:57:09.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 msgr2=0x7f66dc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 0x7f66dc06a870 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6700152620 tx=0x7f66f0007e90 comp rx=0 tx=0).stop 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 msgr2=0x7f6700159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f66e80af040 tx=0x7f66e8019270 comp rx=0 tx=0).stop 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 shutdown_connections 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f66dc0683b0 0x7f66dc06a870 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f670015c1d0 0x7f670015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f67000ccd80 0x7f6700159eb0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 --2- 172.21.15.90:0/1478620870 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f670006e950 0x7f6700153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 >> 172.21.15.90:0/1478620870 conn(0x7f67000b1a50 msgr2=0x7f6700157260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 shutdown_connections 2024-03-31T15:57:09.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.235+0000 7f6706800640 1 -- 172.21.15.90:0/1478620870 wait complete. 2024-03-31T15:57:09.248 INFO:tasks.ceph:Scrubbing osd.7 2024-03-31T15:57:09.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-31T15:57:09.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.391+0000 7f294ba00640 1 Processor -- start 2024-03-31T15:57:09.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.391+0000 7f294ba00640 1 -- start start 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f294ba00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29440ca380 0x7f29440cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f294ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 0x7f294406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f294ba00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29440cb0e0 con 0x7f29440ca760 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f294ba00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f29440ceed0 con 0x7f29440ca380 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f294ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f29440ce350 con 0x7f294406d340 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f29440ca380 0x7f29440cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f293be00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f29440ca760 0x7f294415a9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50126/0 (socket says 172.21.15.90:50126) 2024-03-31T15:57:09.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f293be00640 1 -- 172.21.15.90:0/3531801950 learned_addr learned my addr 172.21.15.90:0/3531801950 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:09.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949a00640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 0x7f294406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 -- 172.21.15.90:0/3531801950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 msgr2=0x7f294406d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 0x7f294406d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 -- 172.21.15.90:0/3531801950 >> v1:172.21.15.90:6789/0 conn(0x7f29440ca760 legacy=0x7f294415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 -- 172.21.15.90:0/3531801950 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29440ce630 con 0x7f29440ca380 2024-03-31T15:57:09.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2943e00640 1 -- 172.21.15.90:0/3531801950 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2045834642 0 0) 0x7f29440cb0e0 con 0x7f29440ca760 2024-03-31T15:57:09.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949a00640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 0x7f294406d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:09.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.395+0000 7f2949000640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca380 0x7f29440cdc50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f293400a960 tx=0x7f293401d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f89ccabe192f8c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2943e00640 1 -- 172.21.15.90:0/3531801950 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340b3070 con 0x7f29440ca380 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2943e00640 1 -- 172.21.15.90:0/3531801950 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29340a68a0 con 0x7f29440ca380 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2943e00640 1 -- 172.21.15.90:0/3531801950 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293400b040 con 0x7f29440ca380 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- 172.21.15.90:0/3531801950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca380 msgr2=0x7f29440cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca380 0x7f29440cdc50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f293400a960 tx=0x7f293401d870 comp rx=0 tx=0).stop 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- 172.21.15.90:0/3531801950 shutdown_connections 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f294406d340 0x7f294406d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- 172.21.15.90:0/3531801950 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca380 0x7f29440cdc50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- 172.21.15.90:0/3531801950 >> 172.21.15.90:0/3531801950 conn(0x7f29440b1880 msgr2=0x7f29440b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- 172.21.15.90:0/3531801950 shutdown_connections 2024-03-31T15:57:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- 172.21.15.90:0/3531801950 wait complete. 2024-03-31T15:57:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 Processor -- start 2024-03-31T15:57:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- start start 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f294406d340 0x7f2944149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29440ca380 0x7f2944151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 0x7f2944151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f294406de90 con 0x7f29440ca760 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f294406dd10 con 0x7f294406d340 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f294ba00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f294406e010 con 0x7f29440ca380 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2949a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 0x7f2944151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2949a00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 0x7f2944151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49582/0 (socket says 172.21.15.90:49582) 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2949a00640 1 -- 172.21.15.90:0/2345010629 learned_addr learned my addr 172.21.15.90:0/2345010629 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f2949000640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f294406d340 0x7f2944149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.399+0000 7f293be00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29440ca380 0x7f2944151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 -- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29440ca380 msgr2=0x7f2944151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29440ca380 0x7f2944151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 -- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f294406d340 msgr2=0x7f2944149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f294406d340 0x7f2944149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294415c170 con 0x7f29440ca760 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f293be00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f29440ca380 0x7f2944151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2949a00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 0x7f2944151f80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f293c009fa0 tx=0x7f293c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c01a070 con 0x7f29440ca760 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c003bc0 con 0x7f29440ca760 2024-03-31T15:57:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c004030 con 0x7f29440ca760 2024-03-31T15:57:09.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294414a210 con 0x7f29440ca760 2024-03-31T15:57:09.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294414b400 con 0x7f29440ca760 2024-03-31T15:57:09.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29440686d0 con 0x7f29440ca760 2024-03-31T15:57:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f293c01d070 con 0x7f29440ca760 2024-03-31T15:57:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2920068380 0x7f292006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f293c086ae0 con 0x7f29440ca760 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7f292006e820 0x7f2920070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2920071330 con 0x7f292006e820 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.403+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f293c08b2a0 con 0x7f29440ca760 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.407+0000 7f2949000640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2920068380 0x7f292006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.407+0000 7f293be00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7f292006e820 0x7f2920070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.407+0000 7f2949000640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2920068380 0x7f292006a840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f29340b0000 tx=0x7f29340a8070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.407+0000 7f293be00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7f292006e820 0x7f2920070c60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.407+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== osd.7 v2:172.21.15.102:6809/552876268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2920071330 con 0x7f292006e820 2024-03-31T15:57:09.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 --> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f29440cd670 con 0x7f292006e820 2024-03-31T15:57:09.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f2941600640 1 -- 172.21.15.90:0/2345010629 <== osd.7 v2:172.21.15.102:6809/552876268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f29440cd670 con 0x7f292006e820 2024-03-31T15:57:09.416 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-03-31T15:57:09.416 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-31T15:57:09.416 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7f292006e820 msgr2=0x7f2920070c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6809/552876268,v1:172.21.15.102:6811/552876268] conn(0x7f292006e820 0x7f2920070c60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2920068380 msgr2=0x7f292006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f2920068380 0x7f292006a840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f29340b0000 tx=0x7f29340a8070 comp rx=0 tx=0).stop 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 msgr2=0x7f2944151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 --2- 172.21.15.90:0/2345010629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29440ca760 0x7f2944151f80 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f293c009fa0 tx=0x7f293c00d560 comp rx=0 tx=0).stop 2024-03-31T15:57:09.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f2949a00640 1 -- 172.21.15.90:0/2345010629 reap_dead start 2024-03-31T15:57:09.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 shutdown_connections 2024-03-31T15:57:09.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 >> 172.21.15.90:0/2345010629 conn(0x7f29440b1880 msgr2=0x7f2944071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 shutdown_connections 2024-03-31T15:57:09.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.419+0000 7f294ba00640 1 -- 172.21.15.90:0/2345010629 wait complete. 2024-03-31T15:57:09.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-31T15:57:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 Processor -- start 2024-03-31T15:57:09.569 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 -- start start 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdde81457b0 0x7fdde806e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdde80721d0 0x7fdde8072630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde8145b90 con 0x7fdde815faf0 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fdde80cf5a0 con 0x7fdde80721d0 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fddee400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdde81608b0 con 0x7fdde81457b0 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdde80721d0 0x7fdde8072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.571 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fdde80721d0 0x7fdde8072630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49594/0 (socket says 172.21.15.90:49594) 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 -- 172.21.15.90:0/3418413334 learned_addr learned my addr 172.21.15.90:0/3418413334 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdded000640 1 --2- 172.21.15.90:0/3418413334 >> v2:172.21.15.102:3300/0 conn(0x7fdde81457b0 0x7fdde806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 -- 172.21.15.90:0/3418413334 >> v2:172.21.15.102:3300/0 conn(0x7fdde81457b0 msgr2=0x7fdde806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 --2- 172.21.15.90:0/3418413334 >> v2:172.21.15.102:3300/0 conn(0x7fdde81457b0 0x7fdde806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 -- 172.21.15.90:0/3418413334 >> v1:172.21.15.102:6789/0 conn(0x7fdde815faf0 legacy=0x7fdde80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.572 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.575+0000 7fdddfe00640 1 -- 172.21.15.90:0/3418413334 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde8072c90 con 0x7fdde80721d0 2024-03-31T15:57:09.573 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fdddfe00640 1 --2- 172.21.15.90:0/3418413334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde80721d0 0x7fdde8072630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fdde8067fb0 tx=0x7fddd001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3228d26f654f10 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fdde7e00640 1 -- 172.21.15.90:0/3418413334 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd00b0070 con 0x7fdde80721d0 2024-03-31T15:57:09.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fdde7e00640 1 -- 172.21.15.90:0/3418413334 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd00ab600 con 0x7fdde80721d0 2024-03-31T15:57:09.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fdde7e00640 1 -- 172.21.15.90:0/3418413334 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd00ab9d0 con 0x7fdde80721d0 2024-03-31T15:57:09.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 -- 172.21.15.90:0/3418413334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde80721d0 msgr2=0x7fdde8072630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 --2- 172.21.15.90:0/3418413334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde80721d0 0x7fdde8072630 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fdde8067fb0 tx=0x7fddd001c520 comp rx=0 tx=0).stop 2024-03-31T15:57:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 -- 172.21.15.90:0/3418413334 shutdown_connections 2024-03-31T15:57:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 --2- 172.21.15.90:0/3418413334 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde80721d0 0x7fdde8072630 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 --2- 172.21.15.90:0/3418413334 >> v2:172.21.15.102:3300/0 conn(0x7fdde81457b0 0x7fdde806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 -- 172.21.15.90:0/3418413334 >> 172.21.15.90:0/3418413334 conn(0x7fdde80b2800 msgr2=0x7fdde80b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 -- 172.21.15.90:0/3418413334 shutdown_connections 2024-03-31T15:57:09.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.579+0000 7fddee400640 1 -- 172.21.15.90:0/3418413334 wait complete. 2024-03-31T15:57:09.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 Processor -- start 2024-03-31T15:57:09.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- start start 2024-03-31T15:57:09.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdde80721d0 0x7fdde815dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdde81457b0 0x7fdde815e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdde8160310 con 0x7fdde815faf0 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdde80d0810 con 0x7fdde80721d0 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fdde813c490 con 0x7fdde81457b0 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49608/0 (socket says 172.21.15.90:49608) 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 -- 172.21.15.90:0/918399959 learned_addr learned my addr 172.21.15.90:0/918399959 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fdddfe00640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdde81457b0 0x7fdde815e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fdded000640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdde80721d0 0x7fdde815dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 -- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdde81457b0 msgr2=0x7fdde815e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdde81457b0 0x7fdde815e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 -- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdde80721d0 msgr2=0x7fdde815dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdde80721d0 0x7fdde815dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 -- 172.21.15.90:0/918399959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde813c770 con 0x7fdde815faf0 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddeda00640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c6c0 tx=0x7fdde000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.583 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0003b80 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde0003d20 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde813ca50 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde00052f0 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.583+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde813d010 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde813d210 con 0x7fdde815faf0 2024-03-31T15:57:09.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdde0005490 con 0x7fdde815faf0 2024-03-31T15:57:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fdde5600640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 0x7fddc406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fdde0086750 con 0x7fdde815faf0 2024-03-31T15:57:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fdded000640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 0x7fddc406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.587+0000 7fdded000640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 0x7fddc406a870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdde81451d0 tx=0x7fddd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.591+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fdde813d210 con 0x7fdde815faf0 2024-03-31T15:57:09.740 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fdde8068d00 con 0x7fddc40683b0 2024-03-31T15:57:09.741 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fdde5600640 1 -- 172.21.15.90:0/918399959 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdde8068d00 con 0x7fddc40683b0 2024-03-31T15:57:09.741 INFO:teuthology.orchestra.run.smithi090.stderr:instructed osd(s) 7 to deep-scrub 2024-03-31T15:57:09.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 msgr2=0x7fddc406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 0x7fddc406a870 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fdde81451d0 tx=0x7fddd8008040 comp rx=0 tx=0).stop 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 msgr2=0x7fdde8143e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdde000c6c0 tx=0x7fdde000cb80 comp rx=0 tx=0).stop 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 shutdown_connections 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fddc40683b0 0x7fddc406a870 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdde815faf0 0x7fdde8143e00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fdde81457b0 0x7fdde815e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.744 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 --2- 172.21.15.90:0/918399959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdde80721d0 0x7fdde815dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.747+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 >> 172.21.15.90:0/918399959 conn(0x7fdde80b2800 msgr2=0x7fdde8164f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.751+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 shutdown_connections 2024-03-31T15:57:09.745 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.751+0000 7fddee400640 1 -- 172.21.15.90:0/918399959 wait complete. 2024-03-31T15:57:09.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:57:09.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.903+0000 7fe9c5c00640 1 Processor -- start 2024-03-31T15:57:09.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.903+0000 7fe9c5c00640 1 -- start start 2024-03-31T15:57:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9c5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c01449a0 0x7fe9c015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9c5c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c00b47e0 con 0x7fe9c006bf30 2024-03-31T15:57:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9c5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c00b4ac0 con 0x7fe9c00b3d90 2024-03-31T15:57:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9c5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c015b7d0 con 0x7fe9c01449a0 2024-03-31T15:57:09.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe9c006bf30 0x7fe9c015ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:50130/0 (socket says 172.21.15.90:50130) 2024-03-31T15:57:09.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 -- 172.21.15.90:0/398807996 learned_addr learned my addr 172.21.15.90:0/398807996 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:09.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bea00640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c01449a0 0x7fe9c015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2827830314 0 0) 0x7fe9c00b47e0 con 0x7fe9c006bf30 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9a4005740 con 0x7fe9c006bf30 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342142999 0 0) 0x7fe9c00b4ac0 con 0x7fe9c00b3d90 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9c00b47e0 con 0x7fe9c00b3d90 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1627323513 0 0) 0x7fe9a4005740 con 0x7fe9c006bf30 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c01449a0 msgr2=0x7fe9c015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c01449a0 0x7fe9c015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 >> v1:172.21.15.102:6789/0 conn(0x7fe9c00b3d90 legacy=0x7fe9c006b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c015bab0 con 0x7fe9c006bf30 2024-03-31T15:57:09.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1803724869 0 0) 0x7fe9b8001a00 con 0x7fe9c006bf30 2024-03-31T15:57:09.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 >> v1:172.21.15.90:6789/0 conn(0x7fe9c006bf30 legacy=0x7fe9c015ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9a4005d90 0x7fe9a4006190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9a4006a20 0x7fe9a4006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9a400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c015b7d0 con 0x7fe9a40073e0 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a400e1d0 con 0x7fe9a4006a20 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a400e3d0 con 0x7fe9a4005d90 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9a400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bea00640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9a4005d90 0x7fe9a4006190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9be000640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9a4006a20 0x7fe9a4006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 -- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9a4005d90 msgr2=0x7fe9a4006190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9a4005d90 0x7fe9a4006190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 -- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9a4006a20 msgr2=0x7fe9a4006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:09.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9a4006a20 0x7fe9a4006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.907+0000 7fe9bf400640 1 -- 172.21.15.90:0/398807996 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a4005740 con 0x7fe9a40073e0 2024-03-31T15:57:09.907 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9bf400640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9a400dbe0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00b2c50 tx=0x7fe9b801f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61dd7f4d222cdfe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b800a410 con 0x7fe9a40073e0 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b80aa630 con 0x7fe9a40073e0 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9bd600640 1 -- 172.21.15.90:0/398807996 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80aaa60 con 0x7fe9a40073e0 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 -- 172.21.15.90:0/398807996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 msgr2=0x7fe9a400dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/398807996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9a400dbe0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00b2c50 tx=0x7fe9b801f9e0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9bf400640 1 -- 172.21.15.90:0/398807996 reap_dead start 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 -- 172.21.15.90:0/398807996 shutdown_connections 2024-03-31T15:57:09.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 -- 172.21.15.90:0/398807996 >> 172.21.15.90:0/398807996 conn(0x7fe9c00b1c80 msgr2=0x7fe9c00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 -- 172.21.15.90:0/398807996 shutdown_connections 2024-03-31T15:57:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 -- 172.21.15.90:0/398807996 wait complete. 2024-03-31T15:57:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.911+0000 7fe9c5c00640 1 Processor -- start 2024-03-31T15:57:09.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- start start 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c015a860 0x7fe9c00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9c015acc0 0x7fe9c00cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c00b41f0 con 0x7fe9a40073e0 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c015b450 con 0x7fe9c015acc0 2024-03-31T15:57:09.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c00b4370 con 0x7fe9c015a860 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49614/0 (socket says 172.21.15.90:49614) 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 -- 172.21.15.90:0/1145701665 learned_addr learned my addr 172.21.15.90:0/1145701665 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bf400640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9c015acc0 0x7fe9c00cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9be000640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c015a860 0x7fe9c00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 -- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c015a860 msgr2=0x7fe9c00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c015a860 0x7fe9c00cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 -- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9c015acc0 msgr2=0x7fe9c00cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:09.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9c015acc0 0x7fe9c00cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 -- 172.21.15.90:0/1145701665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c015c3b0 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9bea00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0011890 tx=0x7fe9b0011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b0019070 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b0004240 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000f720 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c015c690 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c015cc50 con 0x7fe9a40073e0 2024-03-31T15:57:09.913 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.915+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c00686d0 con 0x7fe9a40073e0 2024-03-31T15:57:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.919+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe9b001c070 con 0x7fe9a40073e0 2024-03-31T15:57:09.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.919+0000 7fe9b6a00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 0x7fe9a006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.919+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fe9b0086bf0 con 0x7fe9a40073e0 2024-03-31T15:57:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.919+0000 7fe9be000640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 0x7fe9a006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:09.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.919+0000 7fe9be000640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 0x7fe9a006a840 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00ccb70 tx=0x7fe9ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:09.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:09.923+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fe9b0056c30 con 0x7fe9a40073e0 2024-03-31T15:57:10.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9c00b24a0 con 0x7fe9a0068380 2024-03-31T15:57:10.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9b6a00640 1 -- 172.21.15.90:0/1145701665 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13452 (secure 0 0 0) 0x7fe9c00b24a0 con 0x7fe9a0068380 2024-03-31T15:57:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:10.073 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:57:10.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 msgr2=0x7fe9a006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:10.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 0x7fe9a006a840 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe9c00ccb70 tx=0x7fe9ac009040 comp rx=0 tx=0).stop 2024-03-31T15:57:10.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 msgr2=0x7fe9c01e4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:10.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0011890 tx=0x7fe9b0011d50 comp rx=0 tx=0).stop 2024-03-31T15:57:10.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 shutdown_connections 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fe9a0068380 0x7fe9a006a840 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9c015acc0 0x7fe9c00cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fe9c015a860 0x7fe9c00cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 --2- 172.21.15.90:0/1145701665 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe9a40073e0 0x7fe9c01e4ae0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 >> 172.21.15.90:0/1145701665 conn(0x7fe9c00b1c80 msgr2=0x7fe9c00af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 shutdown_connections 2024-03-31T15:57:10.076 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:10.079+0000 7fe9c5c00640 1 -- 172.21.15.90:0/1145701665 wait complete. 2024-03-31T15:57:10.094 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-31T15:57:09.265346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222784,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751936,"statfs":{"total":773094113280,"available":772865982464,"internally_reserved":0,"allocated":6758400,"data_stored":3686056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.100693"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":245,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-31T15:57:01.170939+0000","last_change":"2024-03-31T15:53:37.318524+0000","last_active":"2024-03-31T15:57:01.170939+0000","last_peered":"2024-03-31T15:57:01.170939+0000","last_clean":"2024-03-31T15:57:01.170939+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:57:01.170939+0000","last_undegraded":"2024-03-31T15:57:01.170939+0000","last_fullsized":"2024-03-31T15:57:01.170939+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-31T15:53:35.674647+0000","last_clean_scrub_stamp":"2024-03-31T15:53:35.674647+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T22:50:01.915073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509485,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509486,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509486,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509486,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1290240,"data_stored":903533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28284,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343556,"statfs":{"total":96636764160,"available":96607801344,"internally_reserved":0,"allocated":1290240,"data_stored":903533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:57:10.097 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-31T15:53:35.674647+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=31, tm_hour=15, tm_min=53, tm_sec=35, tm_wday=6, tm_yday=91, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=31, tm_hour=15, tm_min=57, tm_sec=4, tm_wday=6, tm_yday=91, tm_isdst=0) 2024-03-31T15:57:10.097 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-31T15:57:18.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:18.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:30.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-31T15:57:30.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 Processor -- start 2024-03-31T15:57:30.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 -- start start 2024-03-31T15:57:30.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 0x7f5af8162740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af815f780 con 0x7f5af806d5f0 2024-03-31T15:57:30.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5af806daa0 con 0x7f5af815e1b0 2024-03-31T15:57:30.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5afd400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5af81622b0 con 0x7f5af806c7b0 2024-03-31T15:57:30.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af7200640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5af806d5f0 0x7f5af815e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46590/0 (socket says 172.21.15.90:46590) 2024-03-31T15:57:30.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af7200640 1 -- 172.21.15.90:0/2397919422 learned_addr learned my addr 172.21.15.90:0/2397919422 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:30.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5e00640 1 --2- 172.21.15.90:0/2397919422 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af6800640 1 --2- 172.21.15.90:0/2397919422 >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 0x7f5af8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5400640 1 -- 172.21.15.90:0/2397919422 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3340517410 0 0) 0x7f5af815f780 con 0x7f5af806d5f0 2024-03-31T15:57:30.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5400640 1 -- 172.21.15.90:0/2397919422 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5adc003530 con 0x7f5af806d5f0 2024-03-31T15:57:30.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5e00640 1 -- 172.21.15.90:0/2397919422 >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 msgr2=0x7f5af8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:30.248 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5e00640 1 --2- 172.21.15.90:0/2397919422 >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 0x7f5af8162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5e00640 1 -- 172.21.15.90:0/2397919422 >> v1:172.21.15.90:6789/0 conn(0x7f5af806d5f0 legacy=0x7f5af815e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af5e00640 1 -- 172.21.15.90:0/2397919422 --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af815f1c0 con 0x7f5af806c7b0 2024-03-31T15:57:30.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.251+0000 7f5af6800640 1 --2- 172.21.15.90:0/2397919422 >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 0x7f5af8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-31T15:57:30.249 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5af5e00640 1 --2- 172.21.15.90:0/2397919422 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af806d0b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40097c0 tx=0x7f5ae401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7a4e75fb8146f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5af5400640 1 -- 172.21.15.90:0/2397919422 <== mon.2 v2:172.21.15.102:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40b1070 con 0x7f5af806c7b0 2024-03-31T15:57:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5af5400640 1 -- 172.21.15.90:0/2397919422 <== mon.2 v2:172.21.15.102:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40a8050 con 0x7f5af806c7b0 2024-03-31T15:57:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5af5400640 1 -- 172.21.15.90:0/2397919422 <== mon.2 v2:172.21.15.102:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40ac860 con 0x7f5af806c7b0 2024-03-31T15:57:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 -- 172.21.15.90:0/2397919422 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 msgr2=0x7f5af806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 --2- 172.21.15.90:0/2397919422 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af806d0b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40097c0 tx=0x7f5ae401c3e0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 -- 172.21.15.90:0/2397919422 shutdown_connections 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 --2- 172.21.15.90:0/2397919422 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af806d0b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 --2- 172.21.15.90:0/2397919422 >> v2:172.21.15.102:3300/0 conn(0x7f5af815e1b0 0x7f5af8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 -- 172.21.15.90:0/2397919422 >> 172.21.15.90:0/2397919422 conn(0x7f5af809aa40 msgr2=0x7f5af809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 -- 172.21.15.90:0/2397919422 shutdown_connections 2024-03-31T15:57:30.252 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.255+0000 7f5afd400640 1 -- 172.21.15.90:0/2397919422 wait complete. 2024-03-31T15:57:30.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 Processor -- start 2024-03-31T15:57:30.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- start start 2024-03-31T15:57:30.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af81512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 0x7f5af8070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f5af80d2940 con 0x7f5af806d5f0 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5af80d27c0 con 0x7f5af815e1b0 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f5af80d2ac0 con 0x7f5af806c7b0 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42884/0 (socket says 172.21.15.90:42884) 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 -- 172.21.15.90:0/3519692853 learned_addr learned my addr 172.21.15.90:0/3519692853 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:30.255 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af6800640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af81512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af7200640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 0x7f5af8070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 -- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 msgr2=0x7f5af81512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af81512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 -- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 msgr2=0x7f5af8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 0x7f5af8070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 -- 172.21.15.90:0/3519692853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af815cca0 con 0x7f5af806d5f0 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af7200640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 0x7f5af8070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:30.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5af5e00640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5ae401a010 tx=0x7f5ae401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40bc070 con 0x7f5af806d5f0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40acd80 con 0x7f5af806d5f0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af815cf80 con 0x7f5af806d5f0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40b1040 con 0x7f5af806d5f0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.259+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af815d540 con 0x7f5af806d5f0 2024-03-31T15:57:30.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af80686d0 con 0x7f5af806d5f0 2024-03-31T15:57:30.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ae40bf070 con 0x7f5af806d5f0 2024-03-31T15:57:30.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5aeea00640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 0x7f5ad806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f5ae411ff90 con 0x7f5af806d5f0 2024-03-31T15:57:30.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5af6800640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 0x7f5ad806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.259 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.263+0000 7f5af6800640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 0x7f5ad806a840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5af80d19c0 tx=0x7f5ae800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.267+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f5ae4016090 con 0x7f5af806d5f0 2024-03-31T15:57:30.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 --> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5af8169f90 con 0x7f5ad8068380 2024-03-31T15:57:30.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5aeea00640 1 -- 172.21.15.90:0/3519692853 <== mgr.4105 v2:172.21.15.102:6832/158662636 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f5af8169f90 con 0x7f5ad8068380 2024-03-31T15:57:30.417 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:30.418 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-03-31T15:57:30.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 msgr2=0x7f5ad806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 0x7f5ad806a840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5af80d19c0 tx=0x7f5ae800b040 comp rx=0 tx=0).stop 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 msgr2=0x7f5af80aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5ae401a010 tx=0x7f5ae401ac20 comp rx=0 tx=0).stop 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 shutdown_connections 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f5ad8068380 0x7f5ad806a840 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af815e1b0 0x7f5af8070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5af806d5f0 0x7f5af80aeb10 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 --2- 172.21.15.90:0/3519692853 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f5af806c7b0 0x7f5af81512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.423+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 >> 172.21.15.90:0/3519692853 conn(0x7f5af809aa40 msgr2=0x7f5af815e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.427+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 shutdown_connections 2024-03-31T15:57:30.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.427+0000 7f5afd400640 1 -- 172.21.15.90:0/3519692853 wait complete. 2024-03-31T15:57:30.439 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-03-31T15:57:29.269791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":226928,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754747792,"statfs":{"total":773094113280,"available":772861739008,"internally_reserved":0,"allocated":6758400,"data_stored":3686056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002642"},"pg_stats":[{"pgid":"1.0","version":"58'119","reported_seq":256,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-31T15:57:10.272624+0000","last_change":"2024-03-31T15:57:10.272655+0000","last_active":"2024-03-31T15:57:10.272624+0000","last_peered":"2024-03-31T15:57:10.272624+0000","last_clean":"2024-03-31T15:57:10.272624+0000","last_became_active":"2024-03-31T15:53:37.317376+0000","last_became_peered":"2024-03-31T15:53:37.317376+0000","last_unstale":"2024-03-31T15:57:10.272624+0000","last_undegraded":"2024-03-31T15:57:10.272624+0000","last_fullsized":"2024-03-31T15:57:10.272624+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'119","last_scrub_stamp":"2024-03-31T15:57:10.272546+0000","last_deep_scrub":"58'119","last_deep_scrub_stamp":"2024-03-31T15:57:10.272546+0000","last_clean_scrub_stamp":"2024-03-31T15:57:10.272546+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-01T18:55:55.786051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509490,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509490,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509490,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1290240,"data_stored":903533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":696320,"data_stored":313165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28276,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343564,"statfs":{"total":96636764160,"available":96607809536,"internally_reserved":0,"allocated":1290240,"data_stored":903533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-31T15:57:30.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-31T15:57:30.574 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.579+0000 7fa106c00640 1 Processor -- start 2024-03-31T15:57:30.575 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.579+0000 7fa106c00640 1 -- start start 2024-03-31T15:57:30.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa106c00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa10015b040 0x7fa10015b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa106c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1000725a0 0x7fa1000b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.576 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa106c00640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa100072a60 con 0x7fa100071940 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa106c00640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7fa1000ae890 con 0x7fa10015b040 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa106c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa100072d40 con 0x7fa1000725a0 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa10015b040 0x7fa10015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7fa10015b040 0x7fa10015b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42892/0 (socket says 172.21.15.90:42892) 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 -- 172.21.15.90:0/3955394485 learned_addr learned my addr 172.21.15.90:0/3955394485 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ff400640 1 --1- 172.21.15.90:0/3955394485 >> v1:172.21.15.90:6789/0 conn(0x7fa100071940 0x7fa100071e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46600/0 (socket says 172.21.15.90:46600) 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa104c00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1000725a0 0x7fa1000b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 -- 172.21.15.90:0/3955394485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1000725a0 msgr2=0x7fa1000b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:30.577 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1000725a0 0x7fa1000b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 -- 172.21.15.90:0/3955394485 >> v1:172.21.15.90:6789/0 conn(0x7fa100071940 legacy=0x7fa100071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.578 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 -- 172.21.15.90:0/3955394485 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1000aeac0 con 0x7fa10015b040 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.583+0000 7fa0ffe00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa10015b420 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009870 tx=0x7fa0f001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7486b2269c45841 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa0fea00640 1 -- 172.21.15.90:0/3955394485 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00b0070 con 0x7fa10015b040 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa0fea00640 1 -- 172.21.15.90:0/3955394485 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f00ab600 con 0x7fa10015b040 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa0fea00640 1 -- 172.21.15.90:0/3955394485 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f00ab9d0 con 0x7fa10015b040 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- 172.21.15.90:0/3955394485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 msgr2=0x7fa10015b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.580 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa10015b420 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009870 tx=0x7fa0f001c360 comp rx=0 tx=0).stop 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- 172.21.15.90:0/3955394485 shutdown_connections 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa1000725a0 0x7fa1000b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- 172.21.15.90:0/3955394485 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa10015b420 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- 172.21.15.90:0/3955394485 >> 172.21.15.90:0/3955394485 conn(0x7fa1000b18b0 msgr2=0x7fa1000b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- 172.21.15.90:0/3955394485 shutdown_connections 2024-03-31T15:57:30.581 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- 172.21.15.90:0/3955394485 wait complete. 2024-03-31T15:57:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 Processor -- start 2024-03-31T15:57:30.582 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- start start 2024-03-31T15:57:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa100071940 0x7fa100157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1000725a0 0x7fa100142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa10006d870 con 0x7fa10015b040 2024-03-31T15:57:30.584 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1000b49b0 con 0x7fa1000725a0 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa106c00640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7fa10006d9f0 con 0x7fa100071940 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42900/0 (socket says 172.21.15.90:42900) 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 -- 172.21.15.90:0/4252160549 learned_addr learned my addr 172.21.15.90:0/4252160549 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa0ffe00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa100071940 0x7fa100157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa0ff400640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1000725a0 0x7fa100142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 -- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa100071940 msgr2=0x7fa100157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.585 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa100071940 0x7fa100157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 -- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1000725a0 msgr2=0x7fa100142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1000725a0 0x7fa100142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.587+0000 7fa104c00640 1 -- 172.21.15.90:0/4252160549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa100143280 con 0x7fa10015b040 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa104c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007e40 tx=0x7fa0f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8014620 con 0x7fa10015b040 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f8019070 con 0x7fa10015b040 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa100143560 con 0x7fa10015b040 2024-03-31T15:57:30.586 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8014a20 con 0x7fa10015b040 2024-03-31T15:57:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa100143b20 con 0x7fa10015b040 2024-03-31T15:57:30.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa100075880 con 0x7fa10015b040 2024-03-31T15:57:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa0f8014bc0 con 0x7fa10015b040 2024-03-31T15:57:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 0x7fa0e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7fa0f80870c0 con 0x7fa10015b040 2024-03-31T15:57:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.591+0000 7fa0ffe00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 0x7fa0e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.588 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.595+0000 7fa0ffe00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 0x7fa0e006a840 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00ad000 tx=0x7fa0f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.595+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7fa0f80577e0 con 0x7fa10015b040 2024-03-31T15:57:30.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.755+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa1000685f0 con 0x7fa10015b040 2024-03-31T15:57:30.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7fa0f80570e0 con 0x7fa10015b040 2024-03-31T15:57:30.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa0f7e00640 1 -- 172.21.15.90:0/4252160549 <== mon.0 v2:172.21.15.90:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa0f805b490 con 0x7fa10015b040 2024-03-31T15:57:30.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 msgr2=0x7fa0e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 0x7fa0e006a840 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00ad000 tx=0x7fa0f0008040 comp rx=0 tx=0).stop 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 msgr2=0x7fa100142a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007e40 tx=0x7fa0f800e4b0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 shutdown_connections 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7fa0e0068380 0x7fa0e006a840 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa10015b040 0x7fa100142a30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1000725a0 0x7fa100142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 --2- 172.21.15.90:0/4252160549 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7fa100071940 0x7fa100157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 >> 172.21.15.90:0/4252160549 conn(0x7fa1000b18b0 msgr2=0x7fa1000b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 shutdown_connections 2024-03-31T15:57:30.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.759+0000 7fa106c00640 1 -- 172.21.15.90:0/4252160549 wait complete. 2024-03-31T15:57:30.774 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-31T15:57:30.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-31T15:57:30.908 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 Processor -- start 2024-03-31T15:57:30.909 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 -- start start 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f102415e300 0x7f1024161d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 0x7f102406eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1024070a60 con 0x7f1024162280 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 -- --> v2:172.21.15.90:3300/0 -- mon_getmap magic: 0 -- 0x7f102406e3e0 con 0x7f102415e300 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102c200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f10241628b0 con 0x7f102406d210 2024-03-31T15:57:30.910 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1029800640 1 --2- >> v2:172.21.15.90:3300/0 conn(0x7f102415e300 0x7f1024161d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1028e00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f1024162280 0x7f102406cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:46612/0 (socket says 172.21.15.90:46612) 2024-03-31T15:57:30.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1028e00640 1 -- 172.21.15.90:0/1142769514 learned_addr learned my addr 172.21.15.90:0/1142769514 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-31T15:57:30.911 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102a200640 1 --2- 172.21.15.90:0/1142769514 >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 0x7f102406eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1029800640 1 -- 172.21.15.90:0/1142769514 >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 msgr2=0x7f102406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1029800640 1 --2- 172.21.15.90:0/1142769514 >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 0x7f102406eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1029800640 1 -- 172.21.15.90:0/1142769514 >> v1:172.21.15.90:6789/0 conn(0x7f1024162280 legacy=0x7f102406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1029800640 1 -- 172.21.15.90:0/1142769514 --> v2:172.21.15.90:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1024162b90 con 0x7f102415e300 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f102a200640 1 --2- 172.21.15.90:0/1142769514 >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 0x7f102406eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-31T15:57:30.912 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.915+0000 7f1023e00640 1 -- 172.21.15.90:0/1142769514 <== mon.0 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3162149296 0 0) 0x7f1024070a60 con 0x7f1024162280 2024-03-31T15:57:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f1029800640 1 --2- 172.21.15.90:0/1142769514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f1024161d40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1024072cc0 tx=0x7f101801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baeb5c8cf867b4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f1023e00640 1 -- 172.21.15.90:0/1142769514 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180b0070 con 0x7f102415e300 2024-03-31T15:57:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f1023e00640 1 -- 172.21.15.90:0/1142769514 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f10180ab600 con 0x7f102415e300 2024-03-31T15:57:30.914 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f1023e00640 1 -- 172.21.15.90:0/1142769514 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10180ab9f0 con 0x7f102415e300 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 -- 172.21.15.90:0/1142769514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 msgr2=0x7f1024161d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 --2- 172.21.15.90:0/1142769514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f1024161d40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f1024072cc0 tx=0x7f101801c480 comp rx=0 tx=0).stop 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 -- 172.21.15.90:0/1142769514 shutdown_connections 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 --2- 172.21.15.90:0/1142769514 >> v2:172.21.15.102:3300/0 conn(0x7f102406d210 0x7f102406eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 --2- 172.21.15.90:0/1142769514 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f1024161d40 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.915 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 -- 172.21.15.90:0/1142769514 >> 172.21.15.90:0/1142769514 conn(0x7f10240d13a0 msgr2=0x7f10240d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 -- 172.21.15.90:0/1142769514 shutdown_connections 2024-03-31T15:57:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.919+0000 7f102c200640 1 -- 172.21.15.90:0/1142769514 wait complete. 2024-03-31T15:57:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 Processor -- start 2024-03-31T15:57:30.916 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- start start 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 0x7f10240aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 --2- >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1024162280 0x7f102415e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f102406ff80 con 0x7f102415e300 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f102406fe00 con 0x7f102406d210 2024-03-31T15:57:30.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- --> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] -- mon_getmap magic: 0 -- 0x7f1024070100 con 0x7f1024162280 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:42926/0 (socket says 172.21.15.90:42926) 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 -- 172.21.15.90:0/2667511906 learned_addr learned my addr 172.21.15.90:0/2667511906 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1029800640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 0x7f10240aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102a200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1024162280 0x7f102415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 -- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1024162280 msgr2=0x7f102415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1024162280 0x7f102415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 -- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 msgr2=0x7f10240aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:30.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 0x7f10240aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 -- 172.21.15.90:0/2667511906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1024155240 con 0x7f102415e300 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1029800640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 0x7f10240aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1028e00640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f101000c4b0 tx=0x7f101000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010018070 con 0x7f102415e300 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1010007500 con 0x7f102415e300 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1024155520 con 0x7f102415e300 2024-03-31T15:57:30.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010004a50 con 0x7f102415e300 2024-03-31T15:57:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1024155ae0 con 0x7f102415e300 2024-03-31T15:57:30.922 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.923+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10240686d0 con 0x7f102415e300 2024-03-31T15:57:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.927+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f101001b070 con 0x7f102415e300 2024-03-31T15:57:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.927+0000 7f1021600640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-31T15:57:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.927+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5335+0+0 (secure 0 0 0) 0x7f1010085c90 con 0x7f102415e300 2024-03-31T15:57:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.927+0000 7f1029800640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-31T15:57:30.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.927+0000 7f1029800640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 0x7f100406a840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f10180ad000 tx=0x7f1018008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-31T15:57:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:30.931+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179602 (secure 0 0 0) 0x7f1010056290 con 0x7f102415e300 2024-03-31T15:57:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1024161a80 con 0x7f102415e300 2024-03-31T15:57:31.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f1021600640 1 -- 172.21.15.90:0/2667511906 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+3872 (secure 0 0 0) 0x7f1010055b90 con 0x7f102415e300 2024-03-31T15:57:31.113 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-31T15:57:31.113 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/338021984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":338021984},{"type":"v1","addr":"172.21.15.102:6836","nonce":338021984}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5486,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2414870598","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2414870598},{"type":"v1","addr":"172.21.15.102:6837","nonce":2414870598}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":6210,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6834/3744900218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3744900218},{"type":"v1","addr":"172.21.15.90:6834","nonce":3744900218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6835/4061916369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6833","nonce":4061916369},{"type":"v1","addr":"172.21.15.90:6835","nonce":4061916369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[]} 2024-03-31T15:57:31.113 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 29 2024-03-31T15:57:31.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 msgr2=0x7f100406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 0x7f100406a840 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f10180ad000 tx=0x7f1018008040 comp rx=0 tx=0).stop 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 msgr2=0x7f102415a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f101000c4b0 tx=0x7f101000c970 comp rx=0 tx=0).stop 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 shutdown_connections 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:6832/158662636,v1:172.21.15.102:6833/158662636] conn(0x7f1004068380 0x7f100406a840 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] conn(0x7f1024162280 0x7f102415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:31.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f102415e300 0x7f102415a210 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 --2- 172.21.15.90:0/2667511906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f102406d210 0x7f10240aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-31T15:57:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 >> 172.21.15.90:0/2667511906 conn(0x7f10240d13a0 msgr2=0x7f10240add80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-31T15:57:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 shutdown_connections 2024-03-31T15:57:31.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-31T15:57:31.119+0000 7f102c200640 1 -- 172.21.15.90:0/2667511906 wait complete. 2024-03-31T15:57:31.133 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-31T15:57:31.134 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-31T15:57:31.134 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-31T15:57:31.134 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.145 INFO:tasks.ceph.mds.a:Stopped 2024-03-31T15:57:31.145 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-31T15:57:31.145 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.199 INFO:tasks.ceph.mds.c:Stopped 2024-03-31T15:57:31.199 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-31T15:57:31.199 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.214 INFO:tasks.ceph.mds.b:Stopped 2024-03-31T15:57:31.214 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-31T15:57:31.214 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.271 INFO:tasks.ceph.mds.d:Stopped 2024-03-31T15:57:31.271 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-31T15:57:31.272 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-31T15:57:31.272 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.327 INFO:tasks.ceph.osd.0:Stopped 2024-03-31T15:57:31.327 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-31T15:57:31.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.391 INFO:tasks.ceph.osd.1:Stopped 2024-03-31T15:57:31.391 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-31T15:57:31.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.489 INFO:tasks.ceph.osd.2:Stopped 2024-03-31T15:57:31.489 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-31T15:57:31.489 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.549 INFO:tasks.ceph.osd.3:Stopped 2024-03-31T15:57:31.549 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-31T15:57:31.549 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:31.647 INFO:tasks.ceph.osd.4:Stopped 2024-03-31T15:57:31.648 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-31T15:57:31.648 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:35.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-31T15:57:37.750 INFO:tasks.ceph.osd.5:Stopped 2024-03-31T15:57:37.751 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-31T15:57:37.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:37.807 INFO:tasks.ceph.osd.6:Stopped 2024-03-31T15:57:37.808 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-31T15:57:37.808 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:41.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-31T15:57:41.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-31T15:57:43.910 INFO:tasks.ceph.osd.7:Stopped 2024-03-31T15:57:43.910 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-31T15:57:43.911 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-31T15:57:43.911 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:43.965 INFO:tasks.ceph.mgr.y:Stopped 2024-03-31T15:57:43.965 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-31T15:57:43.965 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:44.024 INFO:tasks.ceph.mgr.x:Stopped 2024-03-31T15:57:44.024 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-31T15:57:44.024 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-31T15:57:44.024 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:44.043 INFO:tasks.ceph.mon.a:Stopped 2024-03-31T15:57:44.043 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-31T15:57:44.043 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:44.060 INFO:tasks.ceph.mon.b:Stopped 2024-03-31T15:57:44.060 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-31T15:57:44.060 INFO:teuthology.orchestra.run:waiting for 300 2024-03-31T15:57:44.119 INFO:tasks.ceph.mon.c:Stopped 2024-03-31T15:57:44.120 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-31T15:57:44.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-03-31T15:57:44.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:57:44.155 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-31T15:57:44.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:57:44.446 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-31T15:57:44.687 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:57:44.687 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-31T15:57:44.921 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:57:44.921 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-31T15:57:45.154 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:57:45.155 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-31T15:57:45.278 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:57:45.279 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-31T15:57:45.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:57:45.437 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-31T15:57:45.633 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:57:45.633 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-31T15:57:45.820 INFO:tasks.ceph:Archiving mon data... 2024-03-31T15:57:45.821 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/data/mon.a.tgz 2024-03-31T15:57:45.821 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-31T15:57:45.830 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.CTLgFKsvVy 2024-03-31T15:57:45.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CTLgFKsvVy 2024-03-31T15:57:45.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.CTLgFKsvVy 2024-03-31T15:57:46.041 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.CTLgFKsvVy is 375KB 2024-03-31T15:57:46.106 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.CTLgFKsvVy 2024-03-31T15:57:46.115 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/data/mon.b.tgz 2024-03-31T15:57:46.115 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-31T15:57:46.124 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.W7dgZO8KmR 2024-03-31T15:57:46.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.W7dgZO8KmR 2024-03-31T15:57:46.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-31T15:57:46.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.W7dgZO8KmR 2024-03-31T15:57:46.360 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.W7dgZO8KmR is 488KB 2024-03-31T15:57:46.427 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.W7dgZO8KmR 2024-03-31T15:57:46.435 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/data/mon.c.tgz 2024-03-31T15:57:46.435 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-31T15:57:46.487 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.6IyrbckC6l 2024-03-31T15:57:46.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6IyrbckC6l 2024-03-31T15:57:46.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.6IyrbckC6l 2024-03-31T15:57:46.721 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.6IyrbckC6l is 488KB 2024-03-31T15:57:46.790 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.6IyrbckC6l 2024-03-31T15:57:46.800 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-31T15:57:46.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-31T15:57:46.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-31T15:57:46.875 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-31T15:57:46.911 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-31T15:57:46.911 INFO:tasks.ceph:Archiving crash dumps... 2024-03-31T15:57:46.915 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi090/crash 2024-03-31T15:57:46.916 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-31T15:57:46.939 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi102/crash 2024-03-31T15:57:46.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-31T15:57:46.971 INFO:tasks.ceph:Shutting down logrotate 2024-03-31T15:57:46.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:46.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:47.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:47.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-31T15:57:47.114 INFO:tasks.ceph:Compressing logs... 2024-03-31T15:57:47.114 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T15:57:47.118 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-31T15:57:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36023.log 2024-03-31T15:57:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-03-31T15:57:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39935.log 2024-03-31T15:57:47.150 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36023.log.gz 2024-03-31T15:57:47.150 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33193.log 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37268.log 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39935.log.gz 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35023.log 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.33193.log.gz 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37268.log.gz 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-03-31T15:57:47.151 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35023.log.gz 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34997.log 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34056.log 2024-03-31T15:57:47.152 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34997.log.gz 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33507.log 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33929.log 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34056.log.gz 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33507.log.gz 2024-03-31T15:57:47.159 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38247.log 2024-03-31T15:57:47.160 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33929.log.gz 2024-03-31T15:57:47.160 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34254.log 2024-03-31T15:57:47.160 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-31T15:57:47.161 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38247.log.gz 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34254.log.gz 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-03-31T15:57:47.162 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37687.log 2024-03-31T15:57:47.163 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-31T15:57:47.163 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-31T15:57:47.167 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35919.log 2024-03-31T15:57:47.167 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37687.log.gz 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34945.log 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35919.log.gz 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34945.log: 93.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.35426.log 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34945.log.gz 2024-03-31T15:57:47.168 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35606.log 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.2.35426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39182.log 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35606.log: 91.5% -- replaced with /var/log/ceph/ceph-client.2.35426.log.gz 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39800.log 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35606.log.gz 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39182.log.gz 2024-03-31T15:57:47.169 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34737.log 2024-03-31T15:57:47.170 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39800.log.gz 2024-03-31T15:57:47.170 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36153.log 2024-03-31T15:57:47.170 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gz 2024-03-31T15:57:47.175 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-03-31T15:57:47.175 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34737.log.gz 2024-03-31T15:57:47.175 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-03-31T15:57:47.175 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36153.log.gz 2024-03-31T15:57:47.175 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37817.log 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35788.log 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37817.log.gz 2024-03-31T15:57:47.176 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36447.log 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35788.log: /var/log/ceph/ceph-client.admin.37583.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35788.log.gz 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36447.log.gz 2024-03-31T15:57:47.177 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-03-31T15:57:47.178 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-31T15:57:47.178 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27566.log 2024-03-31T15:57:47.178 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-03-31T15:57:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2024-03-31T15:57:47.181 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz 2024-03-31T15:57:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.27566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42186.log 2024-03-31T15:57:47.181 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27566.log.gz 2024-03-31T15:57:47.181 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37372.log 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42186.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42186.log.gz 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37372.log.gz 2024-03-31T15:57:47.182 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34058.log 2024-03-31T15:57:47.183 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-03-31T15:57:47.183 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-31T15:57:47.184 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40524.log 2024-03-31T15:57:47.184 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34058.log.gz 2024-03-31T15:57:47.184 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35470.log 2024-03-31T15:57:47.185 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-03-31T15:57:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37946.log 2024-03-31T15:57:47.185 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40524.log.gz 2024-03-31T15:57:47.185 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39584.log 2024-03-31T15:57:47.185 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35470.log.gz 2024-03-31T15:57:47.186 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37398.log 2024-03-31T15:57:47.187 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37946.log.gz 2024-03-31T15:57:47.187 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33981.log 2024-03-31T15:57:47.187 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39584.log.gz 2024-03-31T15:57:47.187 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-03-31T15:57:47.187 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37398.log.gz 2024-03-31T15:57:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38474.log 2024-03-31T15:57:47.188 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33981.log.gz 2024-03-31T15:57:47.188 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33141.log 2024-03-31T15:57:47.188 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-03-31T15:57:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-03-31T15:57:47.190 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33115.log 2024-03-31T15:57:47.190 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38474.log.gz 2024-03-31T15:57:47.190 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37609.log 2024-03-31T15:57:47.190 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33141.log.gz 2024-03-31T15:57:47.194 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27767.log 2024-03-31T15:57:47.194 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27743.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27767.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27767.log.gz 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33492.log 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.1.33492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.33611.log 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.1.33492.log.gz 2024-03-31T15:57:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27671.log 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37888.log 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33115.log.gz 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39025.log 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37609.log.gz 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-03-31T15:57:47.197 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37888.log.gz 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42591.log 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39025.log.gz 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27676.log 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37661.log 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42591.log.gz 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.27676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43415.log 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27676.log.gz 2024-03-31T15:57:47.198 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43623.log 2024-03-31T15:57:47.199 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37661.log.gz 2024-03-31T15:57:47.204 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.3.33611.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-31T15:57:47.204 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27671.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-31T15:57:47.204 INFO:teuthology.orchestra.run.smithi102.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.3.33611.log.gz 2024-03-31T15:57:47.204 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27671.log.gz 2024-03-31T15:57:47.204 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27719.log 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27815.log 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27719.log.gz 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr: 92.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27695.log 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27815.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-31T15:57:47.205 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27815.log.gz 2024-03-31T15:57:47.206 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27647.log 2024-03-31T15:57:47.206 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27695.log.gz 2024-03-31T15:57:47.209 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.27647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27647.log.gz 2024-03-31T15:57:47.209 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-31T15:57:47.209 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-31T15:57:47.209 INFO:teuthology.orchestra.run.smithi102.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-31T15:57:47.209 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-31T15:57:47.212 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-31T15:57:47.212 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43415.log.gz 2024-03-31T15:57:47.212 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38630.log 2024-03-31T15:57:47.212 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43623.log.gz 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38526.log 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34061.log 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38630.log.gz 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34250.log 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38526.log.gz 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37055.log 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34061.log.gz 2024-03-31T15:57:47.213 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34685.log 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34250.log.gz 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40550.log 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37055.log.gz 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32574.log 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34685.log.gz 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34789.log 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40550.log.gz 2024-03-31T15:57:47.214 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32574.log.gz 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32888.log 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34789.log.gz 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42832.log 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36740.log 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32888.log.gz 2024-03-31T15:57:47.215 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42936.log 2024-03-31T15:57:47.216 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42832.log.gz 2024-03-31T15:57:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33877.log 2024-03-31T15:57:47.216 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36740.log.gz 2024-03-31T15:57:47.216 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38195.log 2024-03-31T15:57:47.216 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42936.log.gz 2024-03-31T15:57:47.217 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.b.log: 91.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-31T15:57:47.217 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27791.log 2024-03-31T15:57:47.217 INFO:teuthology.orchestra.run.smithi102.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-31T15:57:47.218 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.27791.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27791.log.gz 2024-03-31T15:57:47.218 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-31T15:57:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33877.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33877.log.gz -5 2024-03-31T15:57:47.222 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2024-03-31T15:57:47.222 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36950.log 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38195.log.gz 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37739.log 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37862.log 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36950.log.gz 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37242.log 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37739.log.gz 2024-03-31T15:57:47.223 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37862.log.gz 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37242.log.gz 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39691.log 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-03-31T15:57:47.224 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38422.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-31T15:57:47.225 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-03-31T15:57:47.225 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43181.log 2024-03-31T15:57:47.225 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39691.log.gz 2024-03-31T15:57:47.225 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2024-03-31T15:57:47.225 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38143.log 2024-03-31T15:57:47.228 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43181.log.gz 2024-03-31T15:57:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33089.log 2024-03-31T15:57:47.228 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2024-03-31T15:57:47.228 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43129.log 2024-03-31T15:57:47.228 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38143.log.gz 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33089.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33089.log.gz 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34919.log 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43129.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22758.log 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43129.log.gz 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34459.log 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34919.log.gz 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.tmp-client.admin.22758.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22758.log.gz 2024-03-31T15:57:47.229 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35997.log 2024-03-31T15:57:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2024-03-31T15:57:47.231 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34459.log: /var/log/ceph/ceph-client.admin.35997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38656.log 2024-03-31T15:57:47.231 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34459.log.gz 2024-03-31T15:57:47.231 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35997.log.gz 2024-03-31T15:57:47.232 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41157.log 2024-03-31T15:57:47.232 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz 2024-03-31T15:57:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38656.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38656.log.gz 2024-03-31T15:57:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42981.log 2024-03-31T15:57:47.233 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33612.log 2024-03-31T15:57:47.233 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41157.log.gz 2024-03-31T15:57:47.234 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40779.log 2024-03-31T15:57:47.234 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42981.log.gz 2024-03-31T15:57:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33612.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33612.log.gz 2024-03-31T15:57:47.235 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2024-03-31T15:57:47.236 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34504.log 2024-03-31T15:57:47.236 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40779.log.gz 2024-03-31T15:57:47.237 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40420.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-31T15:57:47.237 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2024-03-31T15:57:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33481.log 2024-03-31T15:57:47.238 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34504.log.gz 2024-03-31T15:57:47.238 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-03-31T15:57:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33481.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33481.log.gz 2024-03-31T15:57:47.240 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43519.log 2024-03-31T15:57:47.241 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42754.log 2024-03-31T15:57:47.241 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-03-31T15:57:47.242 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-03-31T15:57:47.242 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43519.log.gz 2024-03-31T15:57:47.243 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-03-31T15:57:47.243 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42754.log.gz 2024-03-31T15:57:47.244 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34042.log 2024-03-31T15:57:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41013.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-03-31T15:57:47.245 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-03-31T15:57:47.245 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2024-03-31T15:57:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2024-03-31T15:57:47.246 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34042.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34042.log.gz 2024-03-31T15:57:47.247 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39961.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2024-03-31T15:57:47.247 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42376.log 2024-03-31T15:57:47.248 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-03-31T15:57:47.248 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41473.log: /var/log/ceph/ceph-client.admin.42376.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2024-03-31T15:57:47.249 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-03-31T15:57:47.249 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42376.log.gz 2024-03-31T15:57:47.250 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35580.log 2024-03-31T15:57:47.251 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.4.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-31T15:57:47.251 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-03-31T15:57:47.251 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42728.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2024-03-31T15:57:47.251 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-03-31T15:57:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2024-03-31T15:57:47.252 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32782.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35580.log.gz 2024-03-31T15:57:47.253 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.41976.log -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-03-31T15:57:47.253 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-31T15:57:47.254 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2024-03-31T15:57:47.254 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2024-03-31T15:57:47.255 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41976.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-03-31T15:57:47.255 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36924.log 2024-03-31T15:57:47.256 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-03-31T15:57:47.256 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2024-03-31T15:57:47.257 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36924.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36924.log.gz 2024-03-31T15:57:47.257 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34465.log 2024-03-31T15:57:47.258 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2024-03-31T15:57:47.259 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43059.log: /var/log/ceph/ceph-client.admin.34465.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2024-03-31T15:57:47.259 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34465.log.gz 2024-03-31T15:57:47.259 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-03-31T15:57:47.260 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40196.log 2024-03-31T15:57:47.261 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40498.log: /var/log/ceph/ceph-client.admin.38708.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2024-03-31T15:57:47.261 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-03-31T15:57:47.261 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36290.log 2024-03-31T15:57:47.262 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27785.log 2024-03-31T15:57:47.263 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40196.log.gz 2024-03-31T15:57:47.263 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42806.log 2024-03-31T15:57:47.263 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36290.log.gz 2024-03-31T15:57:47.264 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.27785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-03-31T15:57:47.264 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27785.log.gz 2024-03-31T15:57:47.265 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38329.log 2024-03-31T15:57:47.265 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42806.log.gz 2024-03-31T15:57:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41681.log 2024-03-31T15:57:47.267 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz 2024-03-31T15:57:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38329.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38329.log.gz 2024-03-31T15:57:47.267 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43155.log 2024-03-31T15:57:47.268 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33690.log 2024-03-31T15:57:47.269 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41681.log.gz 2024-03-31T15:57:47.269 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43155.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43155.log.gz 2024-03-31T15:57:47.269 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36766.log 2024-03-31T15:57:47.270 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37479.log 2024-03-31T15:57:47.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33690.log: /var/log/ceph/ceph-client.admin.36766.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33690.log.gz 2024-03-31T15:57:47.271 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36766.log.gz 2024-03-31T15:57:47.271 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38578.log 2024-03-31T15:57:47.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36688.log 2024-03-31T15:57:47.273 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37479.log.gz 2024-03-31T15:57:47.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36475.log 2024-03-31T15:57:47.273 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38578.log.gz 2024-03-31T15:57:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-03-31T15:57:47.275 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36688.log.gz 2024-03-31T15:57:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36475.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36475.log.gz 2024-03-31T15:57:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34255.log 2024-03-31T15:57:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34047.log 2024-03-31T15:57:47.277 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-03-31T15:57:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2024-03-31T15:57:47.277 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34255.log.gz 2024-03-31T15:57:47.278 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40576.log 2024-03-31T15:57:47.279 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34047.log.gz 2024-03-31T15:57:47.279 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35892.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2024-03-31T15:57:47.280 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33063.log 2024-03-31T15:57:47.280 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36818.log 2024-03-31T15:57:47.281 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33063.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40576.log.gz 2024-03-31T15:57:47.282 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41395.log 2024-03-31T15:57:47.282 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33063.log.gz 2024-03-31T15:57:47.282 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36818.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36818.log.gz 2024-03-31T15:57:47.283 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38784.log 2024-03-31T15:57:47.283 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37133.log 2024-03-31T15:57:47.284 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41395.log.gz 2024-03-31T15:57:47.284 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43703.log 2024-03-31T15:57:47.285 INFO:teuthology.orchestra.run.smithi090.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38784.log.gz 2024-03-31T15:57:47.286 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37029.log 2024-03-31T15:57:47.286 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37133.log.gz 2024-03-31T15:57:47.286 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42108.log 2024-03-31T15:57:47.287 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37029.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43703.log.gz 2024-03-31T15:57:47.288 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36584.log 2024-03-31T15:57:47.288 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37029.log.gz 2024-03-31T15:57:47.289 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42108.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.35632.log 2024-03-31T15:57:47.289 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-03-31T15:57:47.289 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.36584.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36395.log 2024-03-31T15:57:47.290 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35632.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36584.log.gz 2024-03-31T15:57:47.291 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34711.log 2024-03-31T15:57:47.291 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35632.log.gz 2024-03-31T15:57:47.292 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39129.log 2024-03-31T15:57:47.292 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36395.log.gz 2024-03-31T15:57:47.292 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-03-31T15:57:47.293 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34711.log.gz 2024-03-31T15:57:47.294 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39400.log 2024-03-31T15:57:47.294 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39129.log.gz 2024-03-31T15:57:47.295 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36714.log 2024-03-31T15:57:47.295 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-03-31T15:57:47.295 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.39400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32631.log 2024-03-31T15:57:47.298 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-03-31T15:57:47.298 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36714.log.gz 2024-03-31T15:57:47.298 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.0.39400.log.gz 2024-03-31T15:57:47.298 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32631.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-31T15:57:47.298 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32631.log.gz 2024-03-31T15:57:47.299 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34841.log 2024-03-31T15:57:47.299 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-03-31T15:57:47.300 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33638.log 2024-03-31T15:57:47.301 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-03-31T15:57:47.301 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34841.log.gz 2024-03-31T15:57:47.302 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-03-31T15:57:47.302 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33638.log.gz 2024-03-31T15:57:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43363.log 2024-03-31T15:57:47.304 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-03-31T15:57:47.304 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-03-31T15:57:47.305 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-03-31T15:57:47.305 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-03-31T15:57:47.306 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43363.log.gz 2024-03-31T15:57:47.307 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2024-03-31T15:57:47.308 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-03-31T15:57:47.308 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38942.log 2024-03-31T15:57:47.308 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-03-31T15:57:47.309 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42565.log 2024-03-31T15:57:47.309 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2024-03-31T15:57:47.310 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-03-31T15:57:47.310 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38942.log.gz 2024-03-31T15:57:47.311 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42884.log 2024-03-31T15:57:47.311 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42565.log.gz 2024-03-31T15:57:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35496.log 2024-03-31T15:57:47.313 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-03-31T15:57:47.313 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35814.log 2024-03-31T15:57:47.314 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42884.log.gz 2024-03-31T15:57:47.315 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42780.log 2024-03-31T15:57:47.315 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35496.log.gz 2024-03-31T15:57:47.316 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2024-03-31T15:57:47.316 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35814.log.gz 2024-03-31T15:57:47.317 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33955.log 2024-03-31T15:57:47.317 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42780.log.gz 2024-03-31T15:57:47.318 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34453.log 2024-03-31T15:57:47.318 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2024-03-31T15:57:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35162.log 2024-03-31T15:57:47.320 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33955.log.gz 2024-03-31T15:57:47.320 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43467.log 2024-03-31T15:57:47.321 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34453.log.gz 2024-03-31T15:57:47.321 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38366.log 2024-03-31T15:57:47.321 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35162.log.gz 2024-03-31T15:57:47.323 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43467.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-31T15:57:47.323 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43467.log.gz 2024-03-31T15:57:47.324 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2024-03-31T15:57:47.324 INFO:teuthology.orchestra.run.smithi090.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38366.log.gz 2024-03-31T15:57:47.325 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-03-31T15:57:47.326 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-03-31T15:57:47.326 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2024-03-31T15:57:47.327 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-31T15:57:47.327 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-03-31T15:57:47.329 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-03-31T15:57:47.329 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-03-31T15:57:47.330 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34055.log 2024-03-31T15:57:47.330 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-31T15:57:47.331 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42032.log 2024-03-31T15:57:47.331 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-03-31T15:57:47.332 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36342.log 2024-03-31T15:57:47.332 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34055.log.gz 2024-03-31T15:57:47.333 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.42032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-03-31T15:57:47.334 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36342.log: 89.5% 90.2% -- replaced with /var/log/ceph/ceph-client.0.42032.log.gz 2024-03-31T15:57:47.335 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36342.log.gz 2024-03-31T15:57:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36501.log 2024-03-31T15:57:47.335 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40961.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-03-31T15:57:47.336 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41631.log 2024-03-31T15:57:47.337 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33011.log 2024-03-31T15:57:47.337 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36501.log.gz 2024-03-31T15:57:47.338 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.41631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41317.log 2024-03-31T15:57:47.340 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33011.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33011.log.gz 2024-03-31T15:57:47.340 INFO:teuthology.orchestra.run.smithi090.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.0.41631.log.gz 2024-03-31T15:57:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34490.log 2024-03-31T15:57:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41317.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41317.log.gz 2024-03-31T15:57:47.341 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-03-31T15:57:47.342 INFO:teuthology.orchestra.run.smithi102.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-31T15:57:47.343 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34815.log 2024-03-31T15:57:47.343 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34490.log.gz 2024-03-31T15:57:47.344 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-03-31T15:57:47.344 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-03-31T15:57:47.345 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43259.log 2024-03-31T15:57:47.345 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34815.log.gz 2024-03-31T15:57:47.347 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36238.log 2024-03-31T15:57:47.347 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-03-31T15:57:47.348 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42481.log 2024-03-31T15:57:47.348 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43259.log.gz 2024-03-31T15:57:47.350 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2024-03-31T15:57:47.351 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36238.log.gz 2024-03-31T15:57:47.352 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-03-31T15:57:47.352 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42481.log.gz 2024-03-31T15:57:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36316.log 2024-03-31T15:57:47.354 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2024-03-31T15:57:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz 2024-03-31T15:57:47.354 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.33903.log 2024-03-31T15:57:47.354 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34269.log 2024-03-31T15:57:47.355 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36316.log.gz 2024-03-31T15:57:47.356 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42676.log 2024-03-31T15:57:47.356 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33903.log.gz 2024-03-31T15:57:47.356 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-03-31T15:57:47.357 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34269.log.gz 2024-03-31T15:57:47.357 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2024-03-31T15:57:47.358 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42676.log.gz 2024-03-31T15:57:47.359 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32983.log 2024-03-31T15:57:47.359 INFO:teuthology.orchestra.run.smithi090.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-03-31T15:57:47.360 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36610.log 2024-03-31T15:57:47.360 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2024-03-31T15:57:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-03-31T15:57:47.363 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32983.log.gz 2024-03-31T15:57:47.363 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36368.log 2024-03-31T15:57:47.363 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36610.log.gz 2024-03-31T15:57:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39051.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-03-31T15:57:47.364 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33851.log 2024-03-31T15:57:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36368.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36368.log.gz 2024-03-31T15:57:47.365 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32916.log 2024-03-31T15:57:47.367 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33429.log 2024-03-31T15:57:47.367 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33851.log.gz 2024-03-31T15:57:47.368 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42539.log 2024-03-31T15:57:47.368 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32916.log.gz 2024-03-31T15:57:47.369 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33429.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33429.log.gz 2024-03-31T15:57:47.370 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32692.log 2024-03-31T15:57:47.371 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42539.log.gz 2024-03-31T15:57:47.371 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36421.log 2024-03-31T15:57:47.374 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32692.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-31T15:57:47.374 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32692.log.gz 2024-03-31T15:57:47.374 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36421.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36421.log.gz 2024-03-31T15:57:47.375 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37424.log 2024-03-31T15:57:47.376 INFO:teuthology.orchestra.run.smithi090.stderr: 85.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-31T15:57:47.377 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42160.log 2024-03-31T15:57:47.378 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-03-31T15:57:47.378 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37424.log.gz 2024-03-31T15:57:47.378 INFO:teuthology.orchestra.run.smithi102.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-31T15:57:47.379 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42160.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42160.log.gz 2024-03-31T15:57:47.379 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32664.log 2024-03-31T15:57:47.381 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35101.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-03-31T15:57:47.382 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37002.log 2024-03-31T15:57:47.382 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32664.log.gz 2024-03-31T15:57:47.383 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35049.log 2024-03-31T15:57:47.384 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40699.log 2024-03-31T15:57:47.384 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37002.log.gz 2024-03-31T15:57:47.385 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35049.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35049.log.gz 2024-03-31T15:57:47.386 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-31T15:57:47.386 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40699.log.gz 2024-03-31T15:57:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39639.log 2024-03-31T15:57:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-03-31T15:57:47.389 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-31T15:57:47.389 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39639.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39639.log.gz 2024-03-31T15:57:47.391 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38301.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-03-31T15:57:47.392 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34266.log 2024-03-31T15:57:47.393 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41918.log 2024-03-31T15:57:47.394 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2024-03-31T15:57:47.394 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34266.log.gz 2024-03-31T15:57:47.395 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37346.log 2024-03-31T15:57:47.395 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41918.log.gz 2024-03-31T15:57:47.396 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42212.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2024-03-31T15:57:47.397 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38037.log 2024-03-31T15:57:47.397 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37346.log.gz 2024-03-31T15:57:47.398 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-03-31T15:57:47.399 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39506.log 2024-03-31T15:57:47.399 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38037.log.gz 2024-03-31T15:57:47.400 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37159.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-03-31T15:57:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39506.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39506.log.gz 2024-03-31T15:57:47.402 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37107.log 2024-03-31T15:57:47.403 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42265.log 2024-03-31T15:57:47.405 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36075.log 2024-03-31T15:57:47.405 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37107.log.gz 2024-03-31T15:57:47.406 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-03-31T15:57:47.406 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42265.log.gz 2024-03-31T15:57:47.407 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36075.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36075.log.gz 2024-03-31T15:57:47.407 INFO:teuthology.orchestra.run.smithi090.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34657.log 2024-03-31T15:57:47.408 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39454.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-03-31T15:57:47.408 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37320.log 2024-03-31T15:57:47.410 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-31T15:57:47.411 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34657.log.gz 2024-03-31T15:57:47.411 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37320.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37320.log.gz 2024-03-31T15:57:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33773.log 2024-03-31T15:57:47.412 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz --verbose 2024-03-31T15:57:47.413 INFO:teuthology.orchestra.run.smithi090.stderr: -- /var/log/ceph/ceph-client.admin.38392.log 2024-03-31T15:57:47.413 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2024-03-31T15:57:47.413 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33773.log.gz 2024-03-31T15:57:47.415 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-03-31T15:57:47.415 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2024-03-31T15:57:47.416 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42239.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2024-03-31T15:57:47.417 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-03-31T15:57:47.417 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-03-31T15:57:47.418 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37505.log 2024-03-31T15:57:47.419 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33825.log 2024-03-31T15:57:47.420 INFO:teuthology.orchestra.run.smithi090.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-03-31T15:57:47.420 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37505.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37505.log.gz 2024-03-31T15:57:47.421 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33825.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33825.log.gz 2024-03-31T15:57:47.422 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-03-31T15:57:47.423 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37914.log 2024-03-31T15:57:47.425 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-03-31T15:57:47.425 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-03-31T15:57:47.425 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37914.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37914.log.gz 2024-03-31T15:57:47.427 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41866.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-03-31T15:57:47.427 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38064.log 2024-03-31T15:57:47.428 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-03-31T15:57:47.430 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-03-31T15:57:47.430 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38064.log.gz 2024-03-31T15:57:47.431 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2024-03-31T15:57:47.431 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-03-31T15:57:47.432 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39987.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-03-31T15:57:47.433 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-03-31T15:57:47.433 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38888.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2024-03-31T15:57:47.433 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38500.log 2024-03-31T15:57:47.435 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-03-31T15:57:47.435 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-03-31T15:57:47.436 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38500.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38500.log.gz 2024-03-31T15:57:47.437 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40896.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-03-31T15:57:47.438 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33664.log 2024-03-31T15:57:47.439 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32953.log 2024-03-31T15:57:47.440 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36527.log 2024-03-31T15:57:47.441 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33664.log.gz 2024-03-31T15:57:47.441 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32953.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32953.log.gz 2024-03-31T15:57:47.442 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36527.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36527.log.gz 2024-03-31T15:57:47.443 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38604.log 2024-03-31T15:57:47.444 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38968.log 2024-03-31T15:57:47.445 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37791.log 2024-03-31T15:57:47.446 INFO:teuthology.orchestra.run.smithi090.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38604.log.gz 2024-03-31T15:57:47.446 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38968.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38968.log.gz 2024-03-31T15:57:47.447 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2024-03-31T15:57:47.448 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37791.log.gz 2024-03-31T15:57:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37765.log 2024-03-31T15:57:47.450 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41447.log 2024-03-31T15:57:47.450 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2024-03-31T15:57:47.451 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37765.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37765.log.gz 2024-03-31T15:57:47.452 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41447.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41447.log.gz 2024-03-31T15:57:47.452 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40342.log 2024-03-31T15:57:47.453 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34007.log 2024-03-31T15:57:47.455 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-03-31T15:57:47.455 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40342.log.gz 2024-03-31T15:57:47.456 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34007.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34007.log.gz 2024-03-31T15:57:47.457 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35075.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz 2024-03-31T15:57:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34464.log 2024-03-31T15:57:47.458 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35331.log 2024-03-31T15:57:47.460 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41421.log 2024-03-31T15:57:47.461 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34464.log.gz 2024-03-31T15:57:47.461 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35331.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35331.log.gz 2024-03-31T15:57:47.463 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41421.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41421.log.gz 2024-03-31T15:57:47.463 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41944.log 2024-03-31T15:57:47.464 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38552.log 2024-03-31T15:57:47.466 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37531.log 2024-03-31T15:57:47.466 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41944.log.gz 2024-03-31T15:57:47.466 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38552.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38552.log.gz 2024-03-31T15:57:47.468 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37531.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37531.log.gz 2024-03-31T15:57:47.468 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33747.log 2024-03-31T15:57:47.469 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42319.log 2024-03-31T15:57:47.471 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41122.log 2024-03-31T15:57:47.471 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33747.log.gz 2024-03-31T15:57:47.472 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42319.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42319.log.gz 2024-03-31T15:57:47.473 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41122.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41122.log.gz 2024-03-31T15:57:47.474 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33167.log 2024-03-31T15:57:47.474 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37713.log 2024-03-31T15:57:47.476 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38840.log 2024-03-31T15:57:47.476 INFO:teuthology.orchestra.run.smithi090.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33167.log.gz 2024-03-31T15:57:47.477 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37713.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37713.log.gz 2024-03-31T15:57:47.480 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.38840.log: 92.1% -- replaced with /var/log/ceph/ceph-client.0.38840.log.gz 2024-03-31T15:57:47.481 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-03-31T15:57:47.482 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2024-03-31T15:57:47.483 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-03-31T15:57:47.484 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-03-31T15:57:47.484 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41787.log 88.4% 2024-03-31T15:57:47.485 INFO:teuthology.orchestra.run.smithi090.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz 2024-03-31T15:57:47.486 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-03-31T15:57:47.486 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gz 2024-03-31T15:57:47.487 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-31T15:57:47.487 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41787.log.gz 2024-03-31T15:57:47.488 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41735.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-03-31T15:57:47.489 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40066.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-03-31T15:57:47.490 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38091.log 2024-03-31T15:57:47.491 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37081.log 2024-03-31T15:57:47.492 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43677.log 2024-03-31T15:57:47.493 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38091.log.gz 2024-03-31T15:57:47.493 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37081.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37081.log.gz 2024-03-31T15:57:47.495 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43677.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43677.log.gz 2024-03-31T15:57:47.495 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-31T15:57:47.496 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35762.log 2024-03-31T15:57:47.498 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36179.log 2024-03-31T15:57:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35762.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35762.log.gz 2024-03-31T15:57:47.499 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36179.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-31T15:57:47.500 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36179.log.gz 2024-03-31T15:57:47.500 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-31T15:57:47.501 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34971.log 2024-03-31T15:57:47.503 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42345.log 2024-03-31T15:57:47.504 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34971.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34971.log.gz 2024-03-31T15:57:47.505 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42345.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42345.log.gz 2024-03-31T15:57:47.506 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36205.log 2024-03-31T15:57:47.507 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33533.log 2024-03-31T15:57:47.508 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36205.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36205.log.gz 2024-03-31T15:57:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33533.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33533.log.gz 2024-03-31T15:57:47.510 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-03-31T15:57:47.511 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43101.log 2024-03-31T15:57:47.512 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39208.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-03-31T15:57:47.513 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43101.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43101.log.gz 2024-03-31T15:57:47.514 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-31T15:57:47.515 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43653.log 2024-03-31T15:57:47.518 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.43653.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43653.log.gz 2024-03-31T15:57:47.518 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43007.log 2024-03-31T15:57:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43007.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43007.log.gz 2024-03-31T15:57:47.521 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37190.log 2024-03-31T15:57:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37190.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37190.log.gz 2024-03-31T15:57:47.524 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39856.log 2024-03-31T15:57:47.528 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33037.log 2024-03-31T15:57:47.529 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.39856.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.39856.log.gz 2024-03-31T15:57:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33037.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33037.log.gz 2024-03-31T15:57:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37216.log 2024-03-31T15:57:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37216.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37216.log.gz 2024-03-31T15:57:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34054.log 2024-03-31T15:57:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34054.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34054.log.gz 2024-03-31T15:57:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-31T15:57:47.539 INFO:teuthology.orchestra.run.smithi090.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-31T15:57:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35684.log 2024-03-31T15:57:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39480.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-31T15:57:47.541 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.35684.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33799.log 2024-03-31T15:57:47.542 INFO:teuthology.orchestra.run.smithi090.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35684.log.gz 2024-03-31T15:57:47.543 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33799.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33799.log.gz 2024-03-31T15:57:47.544 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34867.log 2024-03-31T15:57:47.545 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32860.log 2024-03-31T15:57:47.547 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34867.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34867.log.gz 2024-03-31T15:57:47.548 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32860.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32860.log.gz 2024-03-31T15:57:47.548 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35971.log 2024-03-31T15:57:47.549 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36049.log 2024-03-31T15:57:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35971.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35971.log.gz 2024-03-31T15:57:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36049.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36049.log.gz 2024-03-31T15:57:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-03-31T15:57:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32808.log 2024-03-31T15:57:47.556 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39234.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-03-31T15:57:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32808.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32808.log.gz 2024-03-31T15:57:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33586.log 2024-03-31T15:57:47.557 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34052.log 2024-03-31T15:57:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33586.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33586.log.gz 2024-03-31T15:57:47.559 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-03-31T15:57:47.561 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34052.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34052.log.gz 2024-03-31T15:57:47.562 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5/var/log/ceph/ceph-client.admin.39532.log: --verbose -- /var/log/ceph/ceph-client.admin.42702.log 2024-03-31T15:57:47.562 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-03-31T15:57:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42702.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42702.log.gz 2024-03-31T15:57:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43571.log 2024-03-31T15:57:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33455.log 2024-03-31T15:57:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43571.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43571.log.gz 2024-03-31T15:57:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33455.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33455.log.gz 2024-03-31T15:57:47.569 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43033.log 2024-03-31T15:57:47.570 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36636.log 2024-03-31T15:57:47.571 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43033.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43033.log.gz 2024-03-31T15:57:47.572 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36636.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36636.log.gz 2024-03-31T15:57:47.573 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2024-03-31T15:57:47.574 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42291.log 2024-03-31T15:57:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41343.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2024-03-31T15:57:47.576 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42291.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42291.log.gz 2024-03-31T15:57:47.577 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41209.log 2024-03-31T15:57:47.578 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2024-03-31T15:57:47.579 INFO:teuthology.orchestra.run.smithi090.stderr: 91.5%/var/log/ceph/ceph-client.admin.41209.log: -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-31T15:57:47.579 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41209.log.gz 2024-03-31T15:57:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43493.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2024-03-31T15:57:47.581 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33716.log 2024-03-31T15:57:47.582 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-03-31T15:57:47.584 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-31T15:57:47.584 INFO:teuthology.orchestra.run.smithi090.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33716.log.gz 2024-03-31T15:57:47.585 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37635.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-03-31T15:57:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40673.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-31T15:57:47.586 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35945.log 2024-03-31T15:57:47.587 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34763.log 2024-03-31T15:57:47.589 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2024-03-31T15:57:47.589 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35945.log.gz 2024-03-31T15:57:47.590 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34763.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34763.log.gz 2024-03-31T15:57:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40394.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2024-03-31T15:57:47.591 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43597.log 2024-03-31T15:57:47.592 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2024-03-31T15:57:47.594 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-03-31T15:57:47.594 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43597.log.gz 2024-03-31T15:57:47.595 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36101.log 2024-03-31T15:57:47.595 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2024-03-31T15:57:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39774.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-03-31T15:57:47.596 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35287.log 2024-03-31T15:57:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36101.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36101.log.gz 2024-03-31T15:57:47.597 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-03-31T15:57:47.598 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.35287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37294.log 2024-03-31T15:57:47.599 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40805.log: 90.5% -- replaced with /var/log/ceph/ceph-client.0.35287.log.gz 2024-03-31T15:57:47.600 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39665.log 2024-03-31T15:57:47.600 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-03-31T15:57:47.601 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-03-31T15:57:47.601 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37294.log.gz 2024-03-31T15:57:47.602 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34449.log 2024-03-31T15:57:47.602 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39665.log.gz 2024-03-31T15:57:47.604 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.39260.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-03-31T15:57:47.605 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36127.log 2024-03-31T15:57:47.605 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34449.log.gz 2024-03-31T15:57:47.606 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2024-03-31T15:57:47.607 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34257.log 2024-03-31T15:57:47.607 INFO:teuthology.orchestra.run.smithi090.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36127.log.gz 2024-03-31T15:57:47.608 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40013.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2024-03-31T15:57:47.609 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34257.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34257.log.gz 2024-03-31T15:57:47.610 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42650.log 2024-03-31T15:57:47.612 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41525.log 2024-03-31T15:57:47.613 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-31T15:57:47.613 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42650.log.gz 2024-03-31T15:57:47.614 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41525.log: /var/log/ceph/ceph-client.admin.41183.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41525.log.gz 2024-03-31T15:57:47.615 INFO:teuthology.orchestra.run.smithi090.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-31T15:57:47.615 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32834.log 2024-03-31T15:57:47.616 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38273.log 2024-03-31T15:57:47.618 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38117.log 2024-03-31T15:57:47.618 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32834.log.gz 2024-03-31T15:57:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38273.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38273.log.gz 2024-03-31T15:57:47.619 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-03-31T15:57:47.620 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38117.log: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-31T15:57:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35554.log 2024-03-31T15:57:47.623 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38117.log.gz 2024-03-31T15:57:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42432.log 2024-03-31T15:57:47.623 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-03-31T15:57:47.623 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-03-31T15:57:47.624 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35554.log.gz 2024-03-31T15:57:47.624 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.0.42432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32754.log 2024-03-31T15:57:47.625 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-03-31T15:57:47.625 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-03-31T15:57:47.625 INFO:teuthology.orchestra.run.smithi090.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.0.42432.log.gz 2024-03-31T15:57:47.626 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.32754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41040.log 2024-03-31T15:57:47.626 INFO:teuthology.orchestra.run.smithi090.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32754.log.gz 2024-03-31T15:57:47.628 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-03-31T15:57:47.629 INFO:teuthology.orchestra.run.smithi090.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-03-31T15:57:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.41040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35736.log 2024-03-31T15:57:47.629 INFO:teuthology.orchestra.run.smithi090.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41040.log.gz 2024-03-31T15:57:47.629 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38448.log: gzip -5 --verbose 88.9% -- -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz /var/log/ceph/ceph-client.admin.35710.log 2024-03-31T15:57:47.629 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-31T15:57:47.630 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37450.log 2024-03-31T15:57:47.630 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35736.log.gz 2024-03-31T15:57:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35710.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35710.log.gz 2024-03-31T15:57:47.631 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35528.log 2024-03-31T15:57:47.632 INFO:teuthology.orchestra.run.smithi090.stderr:gzip/var/log/ceph/ceph-client.admin.37450.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33559.log 2024-03-31T15:57:47.633 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37450.log.gz 2024-03-31T15:57:47.633 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36844.log 2024-03-31T15:57:47.633 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-03-31T15:57:47.634 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43207.log 2024-03-31T15:57:47.635 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33559.log.gz 2024-03-31T15:57:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33371.log 2024-03-31T15:57:47.636 INFO:teuthology.orchestra.run.smithi090.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36844.log.gz 2024-03-31T15:57:47.636 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.43207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-03-31T15:57:47.637 INFO:teuthology.orchestra.run.smithi090.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43207.log.gz 2024-03-31T15:57:47.637 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.33371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36662.log 2024-03-31T15:57:47.639 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.34893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36792.log 2024-03-31T15:57:47.639 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-03-31T15:57:47.639 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.33371.log.gz 2024-03-31T15:57:47.640 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42134.log 2024-03-31T15:57:47.640 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36662.log.gz 2024-03-31T15:57:47.641 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36792.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36792.log.gz -5 2024-03-31T15:57:47.641 INFO:teuthology.orchestra.run.smithi090.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2024-03-31T15:57:47.642 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.42134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36558.log 2024-03-31T15:57:47.642 INFO:teuthology.orchestra.run.smithi090.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42134.log.gz 2024-03-31T15:57:47.643 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.40621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37557.log 2024-03-31T15:57:47.644 INFO:teuthology.orchestra.run.smithi090.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2024-03-31T15:57:47.644 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.36558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-03-31T15:57:47.644 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36558.log.gz 2024-03-31T15:57:47.645 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.37557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-03-31T15:57:47.645 INFO:teuthology.orchestra.run.smithi090.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37557.log.gz 2024-03-31T15:57:47.647 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.38734.log: /var/log/ceph/ceph-client.admin.42081.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-03-31T15:57:47.647 INFO:teuthology.orchestra.run.smithi090.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-03-31T15:57:47.924 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-31T15:57:47.964 INFO:teuthology.orchestra.run.smithi090.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-31T15:57:48.005 INFO:teuthology.orchestra.run.smithi102.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-31T15:57:48.007 INFO:teuthology.orchestra.run.smithi102.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-31T15:57:48.018 INFO:teuthology.orchestra.run.smithi102.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-31T15:57:48.020 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-31T15:57:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.851s 2024-03-31T15:57:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m3.311s 2024-03-31T15:57:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.298s 2024-03-31T15:57:48.066 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-31T15:57:48.120 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-31T15:57:48.154 INFO:teuthology.orchestra.run.smithi090.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-31T15:57:48.157 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-31T15:57:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m1.035s 2024-03-31T15:57:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m3.922s 2024-03-31T15:57:48.157 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.943s 2024-03-31T15:57:48.158 INFO:tasks.ceph:Archiving logs... 2024-03-31T15:57:48.158 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi090/log 2024-03-31T15:57:48.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-31T15:57:49.030 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi102/log 2024-03-31T15:57:49.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-31T15:57:49.485 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-31T15:57:49.494 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-31T15:57:49.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-31T15:57:49.498 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-31T15:57:49.578 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-31T15:57:49.590 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-31T15:57:49.590 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:57:49.594 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-31T15:57:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:57:49.607 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2024-03-31T15:57:49.607 INFO:teuthology.orchestra.run.smithi090.stdout:*hv01.front.sepi 44.4.53.2 2 u 8 64 377 0.072 -2.094 1.045 2024-03-31T15:57:49.608 INFO:teuthology.orchestra.run.smithi090.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.089 +1.250 1.504 2024-03-31T15:57:49.608 INFO:teuthology.orchestra.run.smithi090.stdout:+hv03.front.sepi 74.6.168.72 3 u 14 64 377 0.066 -0.835 0.888 2024-03-31T15:57:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:57:49.610 INFO:teuthology.orchestra.run.smithi102.stdout: remote refid st t when poll reach delay offset jitter 2024-03-31T15:57:49.610 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================== 2024-03-31T15:57:49.610 INFO:teuthology.orchestra.run.smithi102.stdout:+hv01.front.sepi 44.4.53.2 2 u 13 64 377 0.083 -1.634 1.185 2024-03-31T15:57:49.610 INFO:teuthology.orchestra.run.smithi102.stdout:+hv02.front.sepi 63.231.80.2 3 u 66 64 177 0.080 -0.039 1.148 2024-03-31T15:57:49.610 INFO:teuthology.orchestra.run.smithi102.stdout:*hv03.front.sepi 74.6.168.72 3 u 11 64 377 0.086 -1.291 1.133 2024-03-31T15:57:49.611 INFO:teuthology.orchestra.run.smithi102.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-31T15:57:49.611 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-31T15:57:49.621 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-31T15:57:49.622 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-31T15:57:49.634 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-31T15:57:49.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-31T15:57:49.653 INFO:teuthology.task.internal:Duration was 938.454405 seconds 2024-03-31T15:57:49.654 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-31T15:57:49.663 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-31T15:57:49.664 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-31T15:57:49.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-31T15:57:49.765 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-31T15:57:49.766 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-03-31T15:57:49.766 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-31T15:57:49.787 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-31T15:57:49.787 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-31T15:57:49.811 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-31T15:57:49.811 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-31T15:57:49.838 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-31T15:57:49.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-31T15:57:49.892 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-31T15:57:49.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-31T15:57:50.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-31T15:57:50.018 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-31T15:57:50.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-31T15:57:50.039 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-31T15:57:50.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-31T15:57:50.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:57:50.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:57:50.104 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-03-31T15:57:50.122 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-31T15:57:50.140 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:57:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:57:50.170 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-31T15:57:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-31T15:57:50.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-31T15:57:50.202 INFO:teuthology.task.internal:Transferring archived files... 2024-03-31T15:57:50.203 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi090 2024-03-31T15:57:50.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-31T15:57:50.239 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664/remote/smithi102 2024-03-31T15:57:50.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-31T15:57:50.268 INFO:teuthology.task.internal:Removing archive directory... 2024-03-31T15:57:50.268 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-31T15:57:50.277 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-31T15:57:50.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-31T15:57:50.325 INFO:teuthology.task.internal:Not uploading archives. 2024-03-31T15:57:50.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-31T15:57:50.340 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-31T15:57:50.340 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-31T15:57:50.343 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-31T15:57:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: 655427 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 31 15:57 /home/ubuntu/cephtest 2024-03-31T15:57:50.364 INFO:teuthology.orchestra.run.smithi102.stdout: 658118 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 31 15:57 /home/ubuntu/cephtest 2024-03-31T15:57:50.367 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-31T15:57:50.382 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-31T15:57:50.405 INFO:teuthology.nuke:Checking targets against current locks 2024-03-31T15:57:50.432 DEBUG:teuthology.nuke:shortname: smithi090 2024-03-31T15:57:50.432 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T15:57:50.459 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-31T15:57:50.460 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-31T15:57:50.464 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 15:31:39.299439', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKckH9MSjojHRMLXVlBU2myYnNZCRAuBqTRWv9TLz664o+HCyz7XYHhQg/z6xP5TlufTtSBUbEPqjadTUuGjeXM='} 2024-03-31T15:57:50.490 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-30_23:12:08-fs-wip-vshankar-testing-20240330.170739-testing-default-smithi/7631664', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-31 15:31:39.301092', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBb/CxZGUrMOZXbVtwaG8bt0zun6ADwzwAEZXPWOWexVynz8YWDsB3qZv8Z66DgK2HJsZ1/00hqYcnbT8T33ugM='} 2024-03-31T15:57:50.524 INFO:teuthology.orchestra.console.smithi090:Power off 2024-03-31T15:57:50.524 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-31T15:57:50.549 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-03-31T15:57:50.550 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:50.567 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-31T15:57:50.567 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-31T15:57:50.578 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-31T15:57:50.591 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-31T15:57:50.591 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:50.615 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-31T15:57:54.580 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:54.616 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:54.690 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-31T15:57:54.731 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-31T15:57:58.694 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:58.732 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-31T15:57:58.806 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-03-31T15:57:58.846 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-31T15:57:58.907 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-03-31T15:57:58.947 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-31T15:57:59.052 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/pool-perm} duration: 938.4544048309326 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default mon.a-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 mon.b-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b4145c849c554861b3f4c377652b2e95 status: fail success: false 2024-03-31T15:57:59.053 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-31T15:57:59.145 INFO:teuthology.run:FAIL